2025-11-01 22:00:07 [scrapy.utils.log] (PID: 148) INFO: Scrapy 2.12.0 started (bot: catalog_extraction) 2025-11-01 22:00:07 [scrapy.utils.log] (PID: 148) INFO: Versions: lxml 5.3.1.0, libxml2 2.12.9, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 24.11.0, Python 3.11.13 (main, Jun 10 2025, 23:54:42) [GCC 12.2.0], pyOpenSSL 25.0.0 (OpenSSL 3.4.1 11 Feb 2025), cryptography 44.0.2, Platform Linux-6.9.12-x86_64-with-glibc2.36 2025-11-01 22:00:07 [mcmaster_carr] (PID: 148) INFO: Starting extraction spider mcmaster_carr... 2025-11-01 22:00:07 [scrapy.addons] (PID: 148) INFO: Enabled addons: [] 2025-11-01 22:00:07 [py.warnings] (PID: 148) WARNING: /usr/local/lib/python3.11/site-packages/scrapy/utils/request.py:120: ScrapyDeprecationWarning: 'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting. It will be removed in a future version of Scrapy. return cls(crawler) 2025-11-01 22:00:07 [scrapy.extensions.telnet] (PID: 148) INFO: Telnet Password: 86635b47592b17ca 2025-11-01 22:00:07 [py.warnings] (PID: 148) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 22:00:07 [scrapy.middleware] (PID: 148) INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.closespider.CloseSpider', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'spidermon.contrib.scrapy.extensions.Spidermon'] 2025-11-01 22:00:07 [scrapy.crawler] (PID: 148) INFO: Overridden settings: {'BOT_NAME': 'catalog_extraction', 'CONCURRENT_ITEMS': 250, 'CONCURRENT_REQUESTS': 8, 'DOWNLOAD_DELAY': 0.9, 'FEED_EXPORT_ENCODING': 'utf-8', 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_extraction/mcmaster_carr/1d8f8344b76e11f0aeb54200a9fe0102.log', 'LOG_FORMAT': '%(asctime)s [%(name)s] (PID: %(process)d) %(levelname)s: ' '%(message)s', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'catalog_extraction.spiders', 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'RETRY_HTTP_CODES': [500, 502, 503, 504, 522, 524, 408, 429, 420], 'SPIDER_MODULES': ['catalog_extraction.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', 'USER_AGENT': None} 2025-11-01 22:00:08 [scrapy_poet.injection] (PID: 148) INFO: Loading providers: [, , , , , , ] 2025-11-01 22:00:08 [scrapy.middleware] (PID: 148) INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scraping_utils.middlewares.downloaders.ProxyManagerDownloaderMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scraping_utils.middlewares.downloaders.HeadersSpooferDownloaderMiddleware', 'scrapy_poet.InjectionMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_poet.DownloaderStatsMiddleware'] 2025-11-01 22:00:08 [NotFoundHandlerSpiderMiddleware] (PID: 148) INFO: NotFoundHandlerSpiderMiddleware running on PRODUCTION environment. 2025-11-01 22:00:08 [scrapy.middleware] (PID: 148) INFO: Enabled spider middlewares: ['catalog_extraction.middlewares.NotFoundHandlerSpiderMiddleware', 'catalog_extraction.middlewares.FixtureSavingMiddleware', 'scrapy_poet.RetryMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2025-11-01 22:00:08 [scrapy.middleware] (PID: 148) INFO: Enabled item pipelines: ['catalog_extraction.pipelines.DuplicatedSKUsFilterPipeline', 'catalog_extraction.pipelines.DiscontinuedProductsAdjustmentPipeline', 'catalog_extraction.pipelines.PriceRoundingPipeline', 'scraping_utils.pipelines.AttachSupplierPipeline', 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline'] 2025-11-01 22:00:08 [scrapy.core.engine] (PID: 148) INFO: Spider opened 2025-11-01 22:00:08 [scrapy.extensions.closespider] (PID: 148) INFO: Spider will stop when no items are produced after 1800 seconds. 2025-11-01 22:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2025-11-01 22:00:08 [scrapy.extensions.telnet] (PID: 148) INFO: Telnet console listening on 127.0.0.1:6024 2025-11-01 22:00:14 [ProxyManagerDownloaderMiddleware] (PID: 148) INFO: Using brd-customer-hl_13cda1e4-zone-sharedpool_datacenter_proxy as the default proxy for ProxyManagerDownloaderMiddleware. 2025-11-01 22:00:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/diameter~9-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/cable-designation~cl2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/outer-jacket-color~green/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/outer-jacket-color~blue/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/outer-jacket-color~purple/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collet-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collar-bearings/shaft-mount-type~clamp-on&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~9-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/industry-designation~cl2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/collet-size~1-3-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/material~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/outer-insulation-color~green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9983429687264&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=collar%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9983429687264%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002294857%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001541924%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/industry-designation~cl2/ 2025-11-01 22:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3966958597415&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=collets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3966958597415%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003446574%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/outer-insulation-color~blue&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/outer-insulation-color~purple&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/material~plastic/ 2025-11-01 22:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/instrumentation-ball-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/outer-insulation-color~green/ 2025-11-01 22:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3696T19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223696T19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/flammability-rating~ul-2556-ft4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:00:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/outer-insulation-color~blue/ 2025-11-01 22:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3223A481&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223223A481%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/outer-insulation-color~purple/ 2025-11-01 22:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation/for-use-on~grout/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-baskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=instrumentation-ball-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 39 pages (at 39 pages/min), scraped 2 items (at 2 items/min) 2025-11-01 22:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/use-for~boats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation/for-use-on~plastic-1/for-use-on~polyimide/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-clamp-on-structural-framing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/specifications-met~flame-rated-ft4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/collet-size~5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~11-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123120%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/specifications-met~flame-rated-ft4/ 2025-11-01 22:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/for-use-on~grout&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3966958597415&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=collets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3966958597415%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003311994%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-plugs-and-receptacles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-baskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/for-use-on~polyimide-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~18-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~21-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929998%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3257A43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223257A43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/size~11-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/insulation/for-use-on~polyimide-plastic/ 2025-11-01 22:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation/shape~film/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7770A27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227770A27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/collets/size~11-mm/ 2025-11-01 22:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-connectors/number-of-poles~19/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=browse-electrical/type~receptacle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/i-beam-tube-clamps/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/size~18-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/interfaces/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/size~21-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/browse-electrical/type~receptacle/ 2025-11-01 22:01:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/collets/size~18-mm/ 2025-11-01 22:01:58 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:01:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/collets/size~21-mm/ 2025-11-01 22:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/shape~film&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=colored-grease-gun-bands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jobsite-boxes/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inventory-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-i-beam-tube-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 83 pages (at 44 pages/min), scraped 4 items (at 2 items/min) 2025-11-01 22:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inverter-rated-motors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=interfaces&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310985%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-connectors/number-of-poles~19&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=column-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1712501638756&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1712501638756%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001551445%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-jobsite-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-identification-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inventory-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/iron-end-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inverter-rated-motors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-square-blades/length~12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/iron-wing-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6168T13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226168T13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6619637192909&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=combination%20square%20blades&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6619637192909%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001622323%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/drive-size~5-mm-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-mesh-partitions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-partitions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-identification-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~1-11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iron-end-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3236610467807&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=combination%20wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3236610467807%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489912%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iron-wing-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/roller-chain/chain-standard~iso/strand-type~double/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-mesh-partitions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-partitions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-rope-accessories/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 126 pages (at 43 pages/min), scraped 5 items (at 1 items/min) 2025-11-01 22:03:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~1-11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~27-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~no-11/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iso-double-strand-roller-chain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/combination-wrenches/size~1-11-16/ 2025-11-01 22:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-skate-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~no-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jam-nuts/thread-size~1-2-28/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-rope-accessories&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~27-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~no-11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combustion-efficiency-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/combination-wrenches/size~27-mm/ 2025-11-01 22:03:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/combination-wrenches/size~no-11/ 2025-11-01 22:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jam-nuts/thread-size~10-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~no-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-terminals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~1-2-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-tank-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jam-nuts/thread-size~m24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/combination-wrenches/size~no-16/ 2025-11-01 22:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=940728682702&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B940728682702%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334370%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~10-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-anchors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compressed-air-filter%252flubricators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-terminals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:04 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~1-2-28&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:04:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:04:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulkhead-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 172 pages (at 46 pages/min), scraped 5 items (at 0 items/min) 2025-11-01 22:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~m24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=940728682702&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B940728682702%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000139805%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-wire-brooms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jancy-slugger-magnetic-base-drills/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compressed-air-piping-system&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=940728682702&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B940728682702%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334859%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:18 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~10-32&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:04:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-anchors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:28 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~m24&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compressed-gas-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jancy-slugger-magnetic-base-drills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stencil-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stellite-cobalt-6b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-saws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jars/capacity~5-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compression-lug-terminals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=step-down-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-screws/length~160-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jars/material~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jaws/vise-type~trailer-hitch/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=step-drills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jic-enclosures/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-saws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jars/capacity-weight-~5-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jig-springs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:05:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/jars/capacity-weight-~5-gal-/ 2025-11-01 22:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 220 pages (at 48 pages/min), scraped 5 items (at 0 items/min) 2025-11-01 22:05:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-screws/length~160mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jars/material~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jaws/vise-type~trailer-hitch&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/joints/connects-to~tube-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jic-enclosures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/knives/for-use-on~wood-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=concentration-meters/for-testing~specific-gravity&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jig-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wood-screws/length~160mm/ 2025-11-01 22:05:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/jars/material~stainless-steel/ 2025-11-01 22:05:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/joints/fitting-connection~press-fit/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979038566398&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979038566398%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001978334%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6380903578127&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=concentration%20meters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6380903578127%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002052081%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=joints/connection-type~tube&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=woodworking-knives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:32 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=concentration-meters/for-testing~specific-gravity&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=concrete-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=joints/connection-type~press-fit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979038566398&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979038566398%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001978334%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/joints/connection-type~tube/ 2025-11-01 22:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stereoscopes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sterile-vials&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sterile-gloves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/joints/connection-type~press-fit/ 2025-11-01 22:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/junction-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/material~jute/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/kanban-bins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sterilization-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979038566398&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979038566398%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001978334%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/workbenches/adjustability~adjustable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=junction-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conductive-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 271 pages (at 51 pages/min), scraped 5 items (at 0 items/min) 2025-11-01 22:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jute-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kanban-bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stick-electrodes/diameter~7-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stickers/message~push&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:11 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sticky-back-cable-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9938597167718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stick%20electrodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9938597167718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000157087%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951892007380&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stickers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951892007380%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002260147%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conduit/thread-size~5-16-24-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=workbenches/adjustability~adjustable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/workstation-lights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9062N117&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229062N117%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8316204383934&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8316204383934%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001485466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/woven-wire-fencing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stirring-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit/thread-size~pg-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stock/material~petg&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9938597167718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stick%20electrodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9938597167718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000157087%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit/thread-size~5-16-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=workstation-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5860541886895&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conduit&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5860541886895%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454937%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995947086760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stock&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995947086760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001549885%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/conduit/thread-size~5-16-24/ 2025-11-01 22:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~0025/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=woven-wire-fencing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=75145K51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2275145K51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9938597167718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stick%20electrodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9938597167718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000157087%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit-compression-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~0536/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 320 pages (at 49 pages/min), scraped 7 items (at 2 items/min) 2025-11-01 22:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stones/abrasive-material~boron-carbide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~0794/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~0025&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-style~slotted/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stones/color~blue&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~1043j/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951022873847&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stones&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951022873847%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003178813%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:17 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:07:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~104am/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401614%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951022873847&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stones&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951022873847%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001527696%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~0536&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:27 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stones/abrasive-material~boron-carbide&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrench-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~0794&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/for-drive-style~slotted&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4805A182&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224805A182%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400806%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~1043j&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~104am&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400967%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wrenches/for-drive-style~slotted/ 2025-11-01 22:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/key-number~1043j/ 2025-11-01 22:07:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:07:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/individual-set~set&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~boric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/key-number~104am/ 2025-11-01 22:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~grain-alcohol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4805A182&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224805A182%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:07:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wrenches/individual-set~set/ 2025-11-01 22:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~2882/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964962%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179313%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~hydrogen-bromide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~hydrogen-iodide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/thread-type~npsc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 369 pages (at 49 pages/min), scraped 7 items (at 0 items/min) 2025-11-01 22:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967470%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4805A182&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224805A182%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4983N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224983N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967215%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121647%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~130-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strainers/fitting-connection~barbed-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~2882&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=79215A716&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2279215A716%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=79215A716&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2279215A716%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=46685K261&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2246685K261%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9758266108659&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=strainers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9758266108659%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002106927%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401785%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:08:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~20-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:08:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/strainers/for-use-with~sodium-hypochlorite-bleach/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3764K53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223764K53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002128503%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/write-on-tags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wyes/fitting-connection~loc-line/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~200-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strainers/for-use-with~grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4420K55&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224420K55%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~25-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002080096%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~31-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strainers/for-use-with~bleach&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=write-on-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9758266108659&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=strainers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9758266108659%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968046%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wyes/connection-style~loc-line&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002109879%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002151458%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4439T38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224439T38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/strainers/for-use-with~bleach/ 2025-11-01 22:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 421 pages (at 52 pages/min), scraped 13 items (at 6 items/min) 2025-11-01 22:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98355K833&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298355K833%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~4-5-gal/capacity~4-1-2-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wyes/connection-style~loc-line/ 2025-11-01 22:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4121T3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224121T3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hazard-lights/bulb-type~xenon/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/xl-series-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002139150%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079035%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/y-splitters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~4-7-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/yarn/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:09:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strapping-crimpers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002117016%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=xenon-hazard-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~1-25-gal/capacity~1-1-4-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=xl-series-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~1-2-13&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~mc45/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=y-splitters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:44 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~4-7-ml&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=yarn&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~p120/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~p129/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~1-1-4-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:09:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~p175/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~mc45&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~350-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=straps/material~polyester-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~1-1-4-gal-/ 2025-11-01 22:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 465 pages (at 44 pages/min), scraped 16 items (at 3 items/min) 2025-11-01 22:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400609%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-ties/overall-length~1-15-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~p120&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9733419879211&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=straps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9733419879211%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511477%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~p129&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-ties/overall-length~1-7-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~p175&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stretch-wrap/material~polyethylene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~350-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1009N81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221009N81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400602%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400501%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8246808234932&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stretch%20wrap&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8246808234932%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510611%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~22-qt/capacity~5-5-gal/capacity~5-1-2-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~1-15-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~350-gal-/ 2025-11-01 22:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~1-7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~sc8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-ties/length~10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-ties/length~12-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001617834%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~55-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~sm866/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001625480%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~5-1-2-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:52 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~1-15-16&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:55 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~1-7-16&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:10:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~5-1-2-gal-/ 2025-11-01 22:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~y1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~sc8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~12-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=string-line&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~55-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~sm866&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 510 pages (at 45 pages/min), scraped 17 items (at 1 items/min) 2025-11-01 22:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/key-number~sc8/ 2025-11-01 22:11:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001625651%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001628138%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~55-gal-/ 2025-11-01 22:11:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400479%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stroke-counters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~y1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-ties/length~5-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=contoured-surface-scrapers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/material~rubber-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/key-number~y1/ 2025-11-01 22:11:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/width~20-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/width~4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/klip-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~5-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/material~rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-brush-holders/backing-width~7-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-covers/food-industry-standard~fda-compliant-21-cfr-181-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001613826%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/width~20mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/width~4-000mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strut-channel-nuts/thread-size~m6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/material~aluminum/ 2025-11-01 22:11:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/material~rubber/ 2025-11-01 22:11:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=klip-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5546066553624&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20brush%20holders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5546066553624%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000087618%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5922932339759&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20covers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5922932339759%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126120%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:11:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/width~20mm/ 2025-11-01 22:11:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/width~4-000mm/ 2025-11-01 22:12:00 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:12:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/knobs/grip-style~smooth/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9014097180972&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=strut%20channel%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9014097180972%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000148470%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7900T34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227900T34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 557 pages (at 47 pages/min), scraped 17 items (at 0 items/min) 2025-11-01 22:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/knobs/thread-size~m3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4772N21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224772N21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/knobs/thread-size~m6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3259T33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223259T33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-rollers/between-frame-width~21&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/knurls/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knobs/grip-style~smooth&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stubby-combination-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1360744530601&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1360744530601%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000111888%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knobs/thread-size~m3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knobs/thread-size~m6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8323758595347&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8323758595347%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509304%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3259T33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223259T33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knurls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyors/between-frame-width~4-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8323758595347&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8323758595347%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8323758595347&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8323758595347%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000148470%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coolant-dispenser-hose-on-off-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=951457901010&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B951457901010%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000090070%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=subminiature-wedge-base-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lags/screw-size~m5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lags/screw-size~m6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3259T33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223259T33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stick-electrodes/for-joining~copper-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5677K75&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225677K75%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/latch-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/latching-power-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:12:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sump-pump-basins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:13:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 605 pages (at 48 pages/min), scraped 21 items (at 4 items/min) 2025-11-01 22:13:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lathe-soft-jaws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lags/screw-size~m5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lags/screw-size~m6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:11 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:13:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper-based-stick-electrodes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=latch-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=latching-power-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9974560404121&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9974560404121%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009744%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9974560404121&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9974560404121%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000034869%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~007549-40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-series~waterfree/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grips/material~bronze-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lathe-soft-jaws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3111105495877&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cord%20grips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3111105495877%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000050743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~125-001de/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9974560404121&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9974560404121%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009744%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/for-manufacturer-series~waterfree&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=surface-mount-stroboscopes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~007549-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cordless-caulk-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-strips/electrical-protection-type~surge-suppression/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cordless-glue-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lavatory-equipment/for-manufacturer-series~waterfree/ 2025-11-01 22:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002134549%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~125-001de&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9974560404121&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9974560404121%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009744%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=surveillance-cameras&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~238a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121910%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~2998-200-001/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2859K14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222859K14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=surge-suppressor-power-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 655 pages (at 50 pages/min), scraped 21 items (at 0 items/min) 2025-11-01 22:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/computer-connection-type~db15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~a-1037-a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~238a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/material~copper-sulfide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453085%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/nema-type~6-15/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~2998-200-001&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/nema-type~l14-30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002118191%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~a-156-a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511734%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7925K544&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227925K544%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110252%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sweeper-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~a-1037-a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3010K51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223010K51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/nema-style~6-15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/nema-style~l14-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002090637%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~b-2489-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~a-156-a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cords/nema-style~6-15/ 2025-11-01 22:14:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cords/nema-style~l14-30/ 2025-11-01 22:14:55 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~a-1037-a&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:14:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~kn-113/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136919%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/voltage~16v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~p6000-eca-het/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/voltage~480v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/voltage~6v-dc-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~b-2489&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 703 pages (at 48 pages/min), scraped 24 items (at 3 items/min) 2025-11-01 22:15:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553606%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003552951%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/certification~enec-marked&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553639%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lavatory-equipment/manufacturer-model-number~b-2489/ 2025-11-01 22:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~s452-159/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~kn-113&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136919%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~p6000-eca-het&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70235K51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270235K51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9241362370870&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9241362370870%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002276096%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70235K56&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270235K56%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corner-band-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002104141%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002117182%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5839K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225839K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8085T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228085T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~s452-159&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136919%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2931K52&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222931K52%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:40 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~p6000-eca-het&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corrugated-duct&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~s66-001/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/panel-cutout-diameter~1-3-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002112142%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5839K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225839K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/panel-cutout-diameter~1-625/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct-hose/material~cotton-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:15:57 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2931K52&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222931K52%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/panel-cutout-diameter~6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~s66-001&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lead-shields/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/for-panel-cutout-diameter~1-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 750 pages (at 47 pages/min), scraped 27 items (at 3 items/min) 2025-11-01 22:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/for-panel-cutout-diameter~1-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002130427%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/counters/measurement-unit~meters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/switches/for-panel-cutout-diameter~1-3-8/ 2025-11-01 22:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5839K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225839K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2931K52&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222931K52%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/voltage~24v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cotton-duct-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/switches/for-panel-cutout-diameter~1-5-8/ 2025-11-01 22:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/for-panel-cutout-diameter~6-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-shields&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553708%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/markers/for-use-on~leather-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/switches/for-panel-cutout-diameter~6-mm/ 2025-11-01 22:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/led-emergency-lights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=counters/unit-of-measure~meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:34 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:16:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/counters/unit-of-measure~meters/ 2025-11-01 22:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553708%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/leds/light-technology~cfl/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=leather-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/countersinks/screw-size~m5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-head-screws/thread-direction~left-hand/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=led-emergency-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/screw-size~m10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-carrier-mounting-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/fitting-connection~o-ring-face-seal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=983458193638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=countersinks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B983458193638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334357%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002364464%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553708%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=LEDs/light-technology~cfl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/left-hand-threaded-shoulder-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=left-hand-threaded-hex-head-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 800 pages (at 50 pages/min), scraped 27 items (at 0 items/min) 2025-11-01 22:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/for-screw-size~5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:10 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/screw-size~m10&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9948839639468&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9948839639468%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969108%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6407T841&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226407T841%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/countersinks/for-screw-size~5-mm/ 2025-11-01 22:17:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~dust&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~hydrogen-fluoride&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/switching-current~28-amp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=14685K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2214685K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=left-hand-threaded-shoulder-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/levels/thread-size~1-4-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002991259%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~hydrogen-iodide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965331%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/levels/thread-size~1-5-16-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7133N6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227133N6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=79215A515&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2279215A515%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/switching-current~28a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/levers/clamp-type~c/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levels/thread-size~1-4-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/switching-voltage~30v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~wood-chips/for-use-with~wood-shavings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967215%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levels/thread-size~1-5-16-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/switches/switching-current~28a/ 2025-11-01 22:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/pipe-size~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4945834745524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4945834745524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453967%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964993%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984579%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=79215A716&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2279215A716%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4945834745524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4945834745524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455450%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levers/clamp-type~c&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000067115%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/switching-voltage~30v-ac-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 850 pages (at 50 pages/min), scraped 31 items (at 4 items/min) 2025-11-01 22:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-stops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4487K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224487K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3277K89&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223277K89%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~3-8-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978677157349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978677157349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001929614%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lifting-swivels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/switches/switching-voltage~30v-ac-1/ 2025-11-01 22:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454138%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5133A11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225133A11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-base-number~ba15s/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~9-16-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-base-number~e39/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4487K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224487K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=93023A661&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2293023A661%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lifting-swivels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivel-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334596%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-tie-downs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/base-number~ba15s&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivel-plates/material~aluminum-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:49 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~9-16-12&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/base-number~e39&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4487K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224487K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/base-number~ba15s/ 2025-11-01 22:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=48758216288&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swivel%20plates&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B48758216288%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-shape-number~par46/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:18:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/base-number~e39/ 2025-11-01 22:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=craftsman-compatible-garage-door-openers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/material~iron-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 901 pages (at 51 pages/min), scraped 35 items (at 4 items/min) 2025-11-01 22:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1093/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/shape~z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1139/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crimpers/for-use-with~smooth-polyester-strapping&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crossover-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swivels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000147754%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-shape-number~par46&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swivels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002309003%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~15-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8050600728590&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=crimpers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8050600728590%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964857%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002054176%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1093&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1092K26&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221092K26%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/thread-size~m7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1139&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1156/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=20885T27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2220885T27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~2-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8351K641&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228351K641%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~1093/ 2025-11-01 22:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swivels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333500%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/drive-size~15-mm-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~1139/ 2025-11-01 22:19:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~12psb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~11-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2333N263&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222333N263%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/crowfeet/drive-size~15-mm-1/ 2025-11-01 22:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1156&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/drive-size~2-1-8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1813/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=syringes/capacity~0-005-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~1156/ 2025-11-01 22:20:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/crowfeet/drive-size~2-1-8-1/ 2025-11-01 22:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crush-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~12psb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/size~11-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 949 pages (at 48 pages/min), scraped 38 items (at 3 items/min) 2025-11-01 22:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2333N263&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222333N263%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crushers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1134507702014&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=syringes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1134507702014%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002106523%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~12psb/ 2025-11-01 22:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1813&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/crowfeet/size~11-mm/ 2025-11-01 22:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6500T71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226500T71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-tighteners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~192/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~1813/ 2025-11-01 22:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2333N263&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222333N263%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-troughs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cableways&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caged-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~3156ll/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~192&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~2821/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-card-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calibration-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~192/ 2025-11-01 22:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~327/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~69/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~3156ll&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~881/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~2821&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=custom-compression-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slots/bracket-type~corner&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~3156ll/ 2025-11-01 22:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 999 pages (at 50 pages/min), scraped 39 items (at 1 items/min) 2025-11-01 22:21:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~2821/ 2025-11-01 22:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~327&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutting-tweezers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9950050661528&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=t%20slots&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9950050661528%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561555%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~9006-80w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~69&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~881&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-slots/t-slotted-framing-component~track-roller/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-slots/t-slotted-framing-component~tube-holder/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cvd-graphene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~327/ 2025-11-01 22:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=47065T236&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2247065T236%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~69/ 2025-11-01 22:21:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~881/ 2025-11-01 22:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~b1a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~btm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~9006-80w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~epx/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slots/t-slot-framing-component~track-roller&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slots/t-slot-framing-component~tube-holder&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~9006-80w/ 2025-11-01 22:21:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/t-slots/t-slot-framing-component~track-roller/ 2025-11-01 22:21:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:21:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/t-slots/t-slot-framing-component~tube-holder/ 2025-11-01 22:21:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~b1a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slotted-framing-label-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slotted-framing-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~btm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cam-latches/specifications-met~en-45545&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:21:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~epx&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~b1a/ 2025-11-01 22:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~btm/ 2025-11-01 22:22:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:22:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2104451082401&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cam%20latches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2104451082401%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004996493%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~epx/ 2025-11-01 22:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1047 pages (at 48 pages/min), scraped 40 items (at 1 items/min) 2025-11-01 22:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dc-to-dc-converters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/voltage~1-5v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slotted-framing-rails/rail-height~80-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/voltage~2-5v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/voltage~48v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=80625A415&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2280625A415%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=decals/message~push&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t12-light-fixtures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/voltage~7v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9988135953728&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=t%20slotted%20framing%20rails&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9988135953728%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000054582%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=depth-gauges/manufacturer-model-number~236hb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981597435777&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=decals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981597435777%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002260147%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/voltage~1-5v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3136N73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223136N73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8692897390024&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=depth%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8692897390024%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123360%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/voltage~2-5v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/voltage~48v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=80625A415&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2280625A415%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~0-059-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9062N117&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229062N117%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/voltage~7v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003552956%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553741%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003552961%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003552942%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9722T116&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229722T116%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3136N73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223136N73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8692897390024&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=depth%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8692897390024%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123360%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K555&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K555%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K567&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K567%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=80625A415&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2280625A415%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~0-059w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9062N117&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229062N117%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6570T93&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226570T93%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~0-059w/ 2025-11-01 22:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1097 pages (at 50 pages/min), scraped 43 items (at 3 items/min) 2025-11-01 22:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~0-19-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3136N73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223136N73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8692897390024&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=depth%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8692897390024%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123360%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:14 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/development-board-sensor-modules/barcode-symbology~matrix-2-of-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/10-watt-light-bulbs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9062N117&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229062N117%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6570T93&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226570T93%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tabletop-blenders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~0-19w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:23:30 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:23:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=development-board-sensor-modules/barcode-type~matrix-2-of-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~10w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=development-board-sensor-modules/sensor-type~barcode-scanner&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:23:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~0-19w/ 2025-11-01 22:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~6-4-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/125-watt-light-bulbs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~8-25-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6570T93&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226570T93%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/development-board-sensor-modules/barcode-type~matrix-2-of-5/ 2025-11-01 22:23:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~10w/ 2025-11-01 22:23:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9580658925258&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=development%20board%20sensor%20modules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9580658925258%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005412760%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5716N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225716N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~708bcz&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~6-4w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~657w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~125w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~8-25w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:02 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbide-tips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dial%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002141067%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~6-4w/ 2025-11-01 22:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1145 pages (at 48 pages/min), scraped 44 items (at 1 items/min) 2025-11-01 22:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dial%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002074351%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~125w/ 2025-11-01 22:24:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~8-25w/ 2025-11-01 22:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/linear-actuators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/linear-motion-bushings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/linear-mounted-ball-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-fiber-composites&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=linear-actuators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity~11-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=linear-motion-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity~130-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity~14-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diamond-grinding-points&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116267%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=linear-mounted-ball-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002106927%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002141144%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diamond-grit-circular-saw-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1255K64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221255K64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3764K53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223764K53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity~18-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/liners/flexibility~extra-flexible/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity~20-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity~3-1-2-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002135431%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1197 pages (at 52 pages/min), scraped 46 items (at 2 items/min) 2025-11-01 22:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002128503%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity~50-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002139065%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-grinders/for-use-with~cutoff-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/liquid-levels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=liners/flexibility~very-flexible&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4439T15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224439T15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/liquid-level-gauges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity~8-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123926%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8549892642718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=die%20grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8549892642718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002135431%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/liners/flexibility~very-flexible/ 2025-11-01 22:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002082852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4439T33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224439T33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-grinders/head-angle~60&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=liquid-levels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/load-rated-bumpers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=liquid-level-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loaded-lip-seals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8549892642718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=die%20grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8549892642718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002135431%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-spring-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lockable-ball-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=load-rated-bumpers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loaded-lip-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locking-panel-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8549892642718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=die%20grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8549892642718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:00 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~1-2-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/capacity-weight-~55-gal-/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locknut-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lockable-ball-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1245 pages (at 48 pages/min), scraped 48 items (at 2 items/min) 2025-11-01 22:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454167%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locks/for-use-on~sheds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-panel-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8549892642718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=die%20grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8549892642718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/for-use-with~formaldehyde&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locknut-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locks/manufacturer~schlage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=case-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/capacity-weight-~55-gal-/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968108%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locks/material~fabric-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locks/for-use-on~sheds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-078/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971224262213&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971224262213%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930309%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locks/manufacturer~schlage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-246/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tapping-fluids/manufacturer~tap-magic/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locks/material~fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:26:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~5-8-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:26:51 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:26:51 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971224262213&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971224262213%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002261236%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-078&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tap-die-and-drill-bit-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:26:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/locks/material~fabric/ 2025-11-01 22:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-4014/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-caulk/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-246&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:27:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tap-magic-tapping-fluids&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1288 pages (at 43 pages/min), scraped 48 items (at 0 items/min) 2025-11-01 22:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-underwater-epoxy-adhesives/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-e-30cl/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:27:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loop-straps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-4014&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/strip-doors/strip-type~low-temperature/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-caulk&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricant-applicators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=directional-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-underwater-epoxy-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-e-30cl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loop-straps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tapered-roller-bearing-outer-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=low-temperature-strip-doors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricant-applicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/food-industry-standard~nsf-standard/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=disposable-boots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~dlcr2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=distance-measuring-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cement-cutting-drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=center-drills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1333 pages (at 45 pages/min), scraped 48 items (at 0 items/min) 2025-11-01 22:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/specifications-met~nsf-ansi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dock-door-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tape-rulers/manufacturer~lufkin/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dlcr2-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=center-locators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=document-tubes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:28:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lubricants/specifications-met~nsf-ansi/ 2025-11-01 22:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=centrifugal-well-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/taps/thread-size~1-4-28-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/made-to-order-pipe-markers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lufkin-tape-rulers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/made-to-order-spacers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~1-7-8-12-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-edge-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-barricades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~1-4-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/made-to-order-springs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451373%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnet-cups/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=made-to-order-pipe-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~3-4-27&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=made-to-order-spacers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/taps/thread-size~1-4-28/ 2025-11-01 22:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-chip-collectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:28:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451843%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-retractable-belt-barriers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=made-to-order-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1377 pages (at 44 pages/min), scraped 48 items (at 0 items/min) 2025-11-01 22:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-stickers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnet-cups&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~6-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-chip-collectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=doors/material~wood-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-retractable-belt-barriers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~8-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000157155%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnets/finish~anodized/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-stickers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1093630895265&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=doors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1093630895265%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000128377%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~8-36&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454197%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~9-16-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334660%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/makita-die-grinders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mandrels/for-use-with~drill-presses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454579%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=double-bolt-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/manifolds/fitting-connection~quick-disconnect/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnets/finish~anodized&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454197%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/manual-hoists/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3815738403502&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3815738403502%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550239%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=makita-die-grinders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mandrels/for-use-with~drill-presses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manifolds/connection-style~quick-disconnect&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-013/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1427 pages (at 50 pages/min), scraped 48 items (at 0 items/min) 2025-11-01 22:30:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454197%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4498129930775&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4498129930775%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967183%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-035&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=teach-pendant-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/manifolds/connection-style~quick-disconnect/ 2025-11-01 22:30:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/markers/color~clear/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manual-hoists&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529552%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-049&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/markers/color~fluorescent-yellow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-013/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:30 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-035&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-and-belt-pulleys/belt-trade-size~xh&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=markers/color~clear&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548063%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-106&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6843160484390&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chain%20and%20belt%20pulleys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6843160484390%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005438464%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9853367128756&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9853367128756%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001527245%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/masking-tape/color~green-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=markers/color~fluorescent-yellow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-122&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001551033%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tee-nuts/thread-size~7-8-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5732N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225732N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548063%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9853367128756&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9853367128756%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002196512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001554673%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:30:56 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-106&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3487712981410&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tee%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3487712981410%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=masking-tape/color~green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:06 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-122&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/masks/application~cold-protection-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1475 pages (at 48 pages/min), scraped 49 items (at 1 items/min) 2025-11-01 22:31:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=94640A415&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2294640A415%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548063%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=masking-tape/color~orange&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/masking-tape/color~green/ 2025-11-01 22:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=masking-tape/color~light-green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-1559&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2584190231621&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2584190231621%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001526091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2584190231621&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2584190231621%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002202371%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=masks/application~cold-protection&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/fitting-connection~luer-lock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542716%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-1883/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=94640A415&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2294640A415%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/masks/application~cold-protection/ 2025-11-01 22:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tees/for-use-with~resin/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mats/for-use-with~sulfuric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977791246039&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tees&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977791246039%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002362024%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A571&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A571%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-222&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=51525K534&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2251525K534%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529769%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=94640A415&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2294640A415%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/for-use-with~resins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tees/military-specification~ms20825-16j/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-395&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-425&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mats/for-use-with~sulfuric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-1883&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:31:59 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-222&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tees/for-use-with~resins/ 2025-11-01 22:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529591%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001532831%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1629427390652&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1629427390652%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001526577%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1524 pages (at 49 pages/min), scraped 51 items (at 2 items/min) 2025-11-01 22:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mercury-vapor-light-bulbs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/specifications-met~ms20825-16j&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:13 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-425&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A688&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A688%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/made-to-order-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tees/specifications-met~ms20825-16j/ 2025-11-01 22:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/thread-type~universal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529591%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/thread-size~1-4-28-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/specifications-met~din-standard/specifications-met~din-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mercury-vapor-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mesh/mesh-size~300-300/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977791246039&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tees&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977791246039%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000132758%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=telephones/mounting-location~wall&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/shaft-type~barbed/barb-direction~two-way/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002061012%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002090013%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/made-to-order-pins-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7397N64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227397N64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529591%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/thread-size~1-4-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8492923186314&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=telephones&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8492923186314%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002266240%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:45 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/specifications-met~din-standard/specifications-met~din-1&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%22276432973296%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh/mesh-size~300-300&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/dowel-pins/thread-size~1-4-28/ 2025-11-01 22:32:53 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=telephones/mounting-location~wall&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/two-way-barbed-dowel-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=telescoping-tube-structural-framing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4620N111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224620N111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3985461849490&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3985461849490%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002051560%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:02 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:33:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/dowel-pins/two-way-barbed-dowel-pins/ 2025-11-01 22:33:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temperature-controllers/enclosure-rating~ip20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1573 pages (at 49 pages/min), scraped 53 items (at 2 items/min) 2025-11-01 22:33:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temperature-panel-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=85385T114&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2285385T114%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drawers/drawer-release~lift-out&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/material~cobalt-steel/shank-type~morse-taper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9956389503914&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=temperature%20controllers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9956389503914%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000062172%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temperature-switches/environment~outdoor&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-head-cap-screws/screw-features~tension-indicating/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9970799178638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drawers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9970799178638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001987670%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=38615K171&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2238615K171%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=830330883100&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=temperature%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B830330883100%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086967%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temporary-outdoor-shelters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=11965A171&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2211965A171%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/cobalt-steel-morse-taper-shank-drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~0-0110&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~0-48-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tension-indicating-hex-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~1-3-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/cobalt-steel-morse-taper-shank-drill-bits/ 2025-11-01 22:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003250399%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=830330883100&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=temperature%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B830330883100%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086967%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003250700%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509053%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=terminal-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:33:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~1-8-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~18-0-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=830330883100&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=temperature%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B830330883100%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086967%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~19-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509345%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1628 pages (at 55 pages/min), scraped 57 items (at 4 items/min) 2025-11-01 22:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509211%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~21-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003250369%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509023%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~3-55-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~4-15-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002893110%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:21 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=test-weights/weight~10-kg&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509191%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509295%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~90/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1323188756785&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=test%20weights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1323188756785%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002004528%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A268&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A268%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~8-7-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermocouple-terminal-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~y&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermocutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~165-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509227%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~220-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermoelectric-coolers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~90-ga&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1323188756785&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=test%20weights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1323188756785%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002004528%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504766%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/length-class~maintenance/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003250655%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~165mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1678 pages (at 50 pages/min), scraped 58 items (at 1 items/min) 2025-11-01 22:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~220-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1323188756785&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=test%20weights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1323188756785%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002004528%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermostat-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/length~165mm/ 2025-11-01 22:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thin-film-coatings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001584620%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/maintenance-length-drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-152&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-27-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/maintenance-length-drill-bits/ 2025-11-01 22:35:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000069310%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-29&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000050841%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-measuring-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000110045%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-sealant-tape/for-use-with~bromine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~19-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-sealants/for-use-with~acetic-acid-white-vinegar&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-sealants/for-use-with~elemental-fluorines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7596424750350&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealant%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7596424750350%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002963056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:52 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-sealants/for-use-with~hydrochloric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~4-55-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000117952%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~4-8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7597890316846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7597890316846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984513%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7597890316846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7597890316846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6802K22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226802K22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7597890316846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7597890316846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002963068%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1730 pages (at 52 pages/min), scraped 58 items (at 0 items/min) 2025-11-01 22:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=45855K88&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2245855K88%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-sealants/manufacturer-model-number~5117&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~4-15-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~6-9-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~4-55mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=45855K88&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2245855K88%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~4-8mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7597890316846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7597890316846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7597890316846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7597890316846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107007%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000039906%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000132520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/shank-size~4-55mm/ 2025-11-01 22:36:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/shank-size~4-8mm/ 2025-11-01 22:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8149K17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228149K17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-arbors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7597890316846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7597890316846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-plug-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000039906%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/thread-size~7-16-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thredolets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=three-ring-folders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thumb-nuts/material~acetal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7151885265529&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=threaded%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7151885265529%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333517%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~1-11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000039906%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8920298804818&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thumb%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8920298804818%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512278%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~8-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1781 pages (at 51 pages/min), scraped 62 items (at 4 items/min) 2025-11-01 22:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6635333760770&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6635333760770%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001629781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:10 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thumb-nuts/material~acetal&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:37:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torch-collet-bodies/torch-series-number~17&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torch-collet-bodies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6635333760770&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6635333760770%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001598294%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torch-nozzles/tig-torch-nozzle-size~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:17 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~1-11-16&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-arbors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:37:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6999905853066&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tig%20torch%20collet%20bodies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6999905853066%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002115553%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:23 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~8-ft&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2215634484173&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tig%20torch%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2215634484173%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123875%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tire-repair-kits/tire-style~tubeless/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:31 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torch-collet-bodies/torch-series-number~17&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tight-grip-push-in-bumpers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tilt-stages&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=timing-belts/belt-trade-number~10-at5-420&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/molybdenum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tire-spoons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:37:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7222310939460&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=timing%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7222310939460%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281983%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tire-repair-kits/for-tire-style~tubeless&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tires/tire-size~5-70-4-50-8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=toilet-flush-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5863N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225863N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:37:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tire-repair-kits/for-tire-style~tubeless/ 2025-11-01 22:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=titanium-zirconium-molybdenum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1409080599349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tires&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1409080599349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116775%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=toilet-seats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=toilet-tank-replacement-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tongue-and-groove-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1830 pages (at 49 pages/min), scraped 62 items (at 0 items/min) 2025-11-01 22:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tool-holding-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5863N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225863N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tool-mats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tooling-ball-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tooling-component-gripper-pads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torq-set-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torque-gauges/power-source~battery&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torque-seal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torque-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5863N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225863N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torque-stripes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/totes/capacity~330-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4763313652393&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=torque%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4763313652393%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076444%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4457A11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224457A11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:52 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:38:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:38:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=touch-free-button-pushers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-fittings/performance~static-dissipative-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=towing-casters-(capacity-2000-lbs-and-above)&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=totes/capacity-weight-~330-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trash-container-casters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=track-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tray-storage-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/totes/capacity-weight-~330-gal-/ 2025-11-01 22:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1886 pages (at 56 pages/min), scraped 63 items (at 1 items/min) 2025-11-01 22:39:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-repair-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bend-and-stay-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/electrical-properties~static-dissipative&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/flammability-rating~ul-94-v-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/turners-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243614%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=turkey-basters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114827%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=turnbuckles/thread-size~10-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweco-fusion-compatible-contact-tips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/flexibility~bend-and-stay&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezers/tweezer-trade-number~m2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=turner%27s-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3217042704133&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=turnbuckles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3217042704133%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000139805%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=twin-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tubing/flexibility~bend-and-stay/ 2025-11-01 22:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3656929274017&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tweezers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3656929274017%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004248579%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243614%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:53 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezers/tweezer-trade-number~m2&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/u-joints/environment~food-and-beverage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=two-line-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=twist-lock-fasteners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=u-bolts/rubber-cushioned-square-u-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243614%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ultem-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/u-bolts/rubber-cushioned-square-u-bolts/ 2025-11-01 22:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1938 pages (at 52 pages/min), scraped 63 items (at 0 items/min) 2025-11-01 22:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=uniforms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=u-joints/environment-1~food-and-beverage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=universal-protractors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ultra-soft-polyurethane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/u-joints/environment-1~food-and-beverage/ 2025-11-01 22:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=urinal-partitions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:23 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:40:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=utility-knives/blade-type~hot&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=utility-knives/manufacturer-model-number~10-399-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8540872660075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=utility%20knives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8540872660075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003034355%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=utility-sinks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8540872660075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=utility%20knives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8540872660075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089456%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=uv-ovens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~1440-j8m-50&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~159-h3m-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2453A61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222453A61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~1922v277&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4927A12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224927A12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005296778%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~1922v604&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005297508%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~1420-h5m-15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005284709%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283353%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7939K518&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227939K518%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005297309%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~240j6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~390j4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 1993 pages (at 55 pages/min), scraped 65 items (at 2 items/min) 2025-11-01 22:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~3l110&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283505%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7939K578&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227939K578%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282713%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~3l390&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283434%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283353%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:41:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~5l240&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~5l670&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283003%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~5vx1900&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~5vx730&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005284216%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282335%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283753%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283353%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~ax85&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~b93&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~bb60&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281542%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283830%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005284561%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~bx95&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~bx71/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~c48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~c65&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2046 pages (at 53 pages/min), scraped 67 items (at 2 items/min) 2025-11-01 22:42:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283850%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282425%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282838%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~cx144&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~xpa1007&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~xpa1382&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~xpa857&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281183%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283361%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~bx71/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~xpb2840&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005284574%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282590%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~xpz1060&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~xpz1800&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005284258%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283309%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~115&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~123&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~177&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~45&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000168069%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~38-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000059826%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2099 pages (at 53 pages/min), scraped 67 items (at 0 items/min) 2025-11-01 22:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000159892%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000111440%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~48-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~59&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000042900%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~732-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000098031%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~78&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000072406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000075319%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-air-separators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000049328%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valve-core-removers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valve-keys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valve-chain-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/dispenser-type~gravity-flow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002245636%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/for-use-with~potassium-sulfate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/valve-operation~toggle-switch&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:43:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vaporizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=variance-indicators/manufacturer-model-number~2424a-19&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2783K24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222783K24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968123%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001982143%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=variance-indicator-contact-points/manufacturer-model-number~74-106363&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=variance-indicators/manufacturer-model-number~81850&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/variance-indicators/manufacturer-model-number~2776acal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2154 pages (at 55 pages/min), scraped 67 items (at 0 items/min) 2025-11-01 22:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9016360397335&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=variance%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9016360397335%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136578%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=variance-indicators/power-source~battery&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=46315K24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2246315K24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6790T106&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226790T106%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9657484587707&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=variance%20indicator%20contact%20points&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9657484587707%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002139602%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9016360397335&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=variance%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9016360397335%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003448817%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=variance-indicators/manufacturer-model-number~646pka079b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9016360397335&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=variance%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9016360397335%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076444%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/variance-indicators/manufacturer-model-number~646pka079b/ 2025-11-01 22:44:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cloth/material~velvet/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vcr-compatible-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/variance-indicators/manufacturer-model-number~2776acal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/corner-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vents/for-use-with~kerosene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vga-splitters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vise-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=voltage-boosters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=velvet-cloth&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=voltage-to-pressure-transducers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2491040633414&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=vents&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2491040633414%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967061%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vises/corner-clamps-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:44:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:44:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:44:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8126K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228126K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wagon-brake-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7755808933160&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=vises&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7755808933160%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%227774536975162%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/screw-size~m19/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/screw-size~m38/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2200 pages (at 46 pages/min), scraped 71 items (at 4 items/min) 2025-11-01 22:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stools/mounting-location~wall/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/screw-size~m2-6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5017A51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225017A51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washdown-actuators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~an901-10c/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wall-thermometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~an960-c1016/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/for-screw-size~m19&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/for-screw-size~m38&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wall-mount-stools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~an960-c3l/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~an960-c516l/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/for-screw-size~m2-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/for-screw-size~m19/ 2025-11-01 22:45:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/for-screw-size~m38/ 2025-11-01 22:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~an901-10c&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms15795-411/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~an960-c1016&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/for-screw-size~m2-6/ 2025-11-01 22:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms15795-847b/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~an901-10c/ 2025-11-01 22:45:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms15795-805b/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms27183-45/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~an960-c1016/ 2025-11-01 22:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~an960-c3l&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~an960-c516l&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms15795-825/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms15795-411&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~an960-c3l/ 2025-11-01 22:45:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:45:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~an960-c516l/ 2025-11-01 22:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms15795-847b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~din-6798&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms15795-805b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms27183-45&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms15795-411/ 2025-11-01 22:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2244 pages (at 44 pages/min), scraped 72 items (at 1 items/min) 2025-11-01 22:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-cn532r&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms15795-847b/ 2025-11-01 22:46:11 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:46:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002105809%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms15795-805b/ 2025-11-01 22:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~ms15795-825&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms27183-45/ 2025-11-01 22:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003796652%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-f0832p&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:21 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~din-6798&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1515-h12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/specifications-met~ms15795-825/ 2025-11-01 22:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~sae-j314&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:28 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-cn532r&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079683%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-column-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-heat-exchangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002073593%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-hardness-test-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002149848%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:40 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-f0832p&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-metering-drum-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-sprinklers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=waterless-hand-soap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:52 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:46:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:46:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=waterproof-rivets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=way-wipers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=webbing-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wedges/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weld-fillet-gages&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2298 pages (at 54 pages/min), scraped 72 items (at 0 items/min) 2025-11-01 22:47:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weld-flux-chisels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2317539866962&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wedges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2317539866962%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weld-together-framing-rails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-gas-regulators/for-use-with~helium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-cable-ends&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8282146116149&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=welding%20gas%20regulators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8282146116149%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964764%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-rods/aws-material-code~er70s-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2317539866962&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wedges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2317539866962%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=66325A42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2266325A42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-wire/aws-material-code~er308l&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-wire-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8219223485911&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=welding%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8219223485911%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004291833%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=western-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7852200896140&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=welding%20wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7852200896140%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002327035%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheels/wheel-diameter~21-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:53 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-rods/aws-material-code~er70s-7&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:47:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wilkerson-compressed-air-lubricators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2317539866962&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wedges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2317539866962%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:47:58 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-wire/aws-material-code~er308l&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9899648339233&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9899648339233%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001924001%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wind-driven-ventilators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2349 pages (at 51 pages/min), scraped 73 items (at 1 items/min) 2025-11-01 22:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-frosting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-latches/material~steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-opening-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:16 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=windows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wing-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8006903947641&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=window%20latches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8006903947641%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bronze-alloy-316-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wing-turn-draw-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wipes/for-use-on~stone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/100mxl-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/american-wire-gauge-awg~4-0&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/1018-carbon-steel-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/cable-awg~26-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coolants/manufacturer~trico/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2776320989747&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wipes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2776320989747%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930162%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002098053%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-alloy-316-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561273%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/2205-stainless-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:51 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=100mxl-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=1018-carbon-steel-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8219K68&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228219K68%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coolants/manufacturer~trico&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/charging-stations/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2396 pages (at 47 pages/min), scraped 74 items (at 1 items/min) 2025-11-01 22:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5998243833923&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5998243833923%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002251132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=2205-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-ends&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/combination-wrenches/size~5-16/ 2025-11-01 22:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-eyes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-size~1-1-8-5-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1455K25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221455K25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cooling-fans/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/thread-size~m30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=charging-stations&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/3d-printer-filaments/performance~conductive-2/flexibility~flexible/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~1-1-8-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-coolant-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/12-volt-fuel-transfer-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000145706%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cooling-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~m30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-hoists&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bolts/thread-size~1-1-8-5/ 2025-11-01 22:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/conductive-flexible-3d-printer-filaments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334725%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combustible-signs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-coolant-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heli-coil-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/120-degree-gears/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~ax47/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=12-volt-fuel-transfer-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/3d-printer-filaments/conductive-flexible-3d-printer-filaments/ 2025-11-01 22:50:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/die-springs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2439 pages (at 43 pages/min), scraped 75 items (at 1 items/min) 2025-11-01 22:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~18-1016-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/commercially-pure-copper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combustible-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/retaining-ring-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=120-degree-gears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ax47-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~19-04-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000041586%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~18-1016mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-2531/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=commercially-pure-copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retaining-ring-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~elb0610n/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~18-1016mm/ 2025-11-01 22:50:40 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:50:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~19-04mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/copper-alloy-110-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/13a-batteries/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helical-vibration-isolators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~19-04mm/ 2025-11-01 22:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~a19/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~2-071/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-2531&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~elb0610n&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/retractable-water-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:50:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper-alloy-110-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~9-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496562%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002111697%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=13a-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~7207-2rs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2485 pages (at 46 pages/min), scraped 75 items (at 0 items/min) 2025-11-01 22:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070588%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a19-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~2-071&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2267N94&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222267N94%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retractable-water-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyester/backing-type~adhesive/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000161242%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~7207-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-size~11-16-18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chair-levelers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drivers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7197N14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227197N14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/military-specification~ms124670/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-0135&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-compressor-intake-filters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002109914%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-back-polyester-(pet)&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539554%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cap-nuts/thread-size~1-1-4-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~11-16-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/a286-super-alloy-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/military-specifications-met~ms124670&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A298&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A298%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-compressor-intake-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453984%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002109914%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/helicoils/military-specifications-met~ms124670/ 2025-11-01 22:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-condensate-separators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-pipe-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2529 pages (at 44 pages/min), scraped 78 items (at 3 items/min) 2025-11-01 22:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cap-nuts/thread-size~1-1-4-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a286-super-alloy-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/channels/material~polypropylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/finish~powder-coated-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-detectable-sleeves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7896753628184&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7896753628184%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334373%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-4377/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002109914%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-condensate-separators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-pipe-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:26 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cap-nuts/thread-size~1-1-4-12&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:52:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~1-2-10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=channels/material~polypropylene-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/finish~powder-coated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-detectable-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/military-specification~ms122157/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-4377&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:40 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cpvc-pipe-unions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/channels/material~polypropylene-plastic/ 2025-11-01 22:52:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pins/finish~powder-coated/ 2025-11-01 22:52:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-459/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001499837%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/military-specification~ms124653/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-228/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-pipe-locknuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/military-specification~ms124666/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~1-2-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:52:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/specifications-met~ms122-157&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cpvc-pipe-unions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449455%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/helicoils/specifications-met~ms122-157/ 2025-11-01 22:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2571 pages (at 42 pages/min), scraped 78 items (at 0 items/min) 2025-11-01 22:53:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-459&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/specifications-met~ms124-653&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-228&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-pipe-locknuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/copper-nickel-pipe/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/specifications-met~ms124-666&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-reducers/material~bronze-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498319%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/helicoils/specifications-met~ms124-653/ 2025-11-01 22:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-size~3-8-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001534640%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/helicoils/specifications-met~ms124-666/ 2025-11-01 22:53:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/axles/diameter~1-11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89955K258&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289955K258%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:32 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-228&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bronze-pipe-weld-elbows/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hitch-inserts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper-nickel-pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cheaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-pipe-reducers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~3-8-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-size~4-48/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:47 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:53:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~1-4-19-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=axles/diameter~1-11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454138%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-pipe-weld-elbows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/1922v403-v-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-dryers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/axles/diameter~10-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/hitch-inserts-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681008859035&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681008859035%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000024026%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A712&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A712%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~4-48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%226035317718745%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2615 pages (at 44 pages/min), scraped 79 items (at 1 items/min) 2025-11-01 22:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~1-4-19&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4138N26&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224138N26%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334791%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=1922v403-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-dryers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=axles/diameter~10mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2507T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222507T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/dies/thread-size~1-4-19/ 2025-11-01 22:54:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~c87/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/axles/diameter~10mm/ 2025-11-01 22:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/air-exhaust-filters already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~2-5625/od~2-9-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ribbon-cable-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-plates/thickness~11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crane-hoists/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-074&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~0-70-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=c87-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/side-load-threaded-rod-mount-loop-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~2-9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ribbon-cable-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001524051%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/air-exhaust-filters already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9726243375694&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9726243375694%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%222229037755462%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crane-hoists&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~2-9-16/ 2025-11-01 22:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9495772993241&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9495772993241%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001458159%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:54:57 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-074&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~0-7-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3131T52&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223131T52%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~12-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crank-arms/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~13-16-18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2655 pages (at 40 pages/min), scraped 82 items (at 3 items/min) 2025-11-01 22:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9008K83&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229008K83%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~0-7-mm/ 2025-11-01 22:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:13 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drums/capacity~20-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~1-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~12-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-plates/thickness~5-8-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crank-arms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-81/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~13-16-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/material~wood-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334663%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000141432%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drums/capacity-weight-~20-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453971%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~1-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000128377%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drums/capacity-weight-~20-gal-/ 2025-11-01 22:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3283K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223283K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-81&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:55:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~1-1-4/ 2025-11-01 22:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/240h-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/aluminum-plates/thickness~5-8 2025-11-01 22:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/thread-size~m12-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chemical-resistant-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~gya108rrb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/240l-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493402%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:55:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/caps/pipe-size~1-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/a-frame-trucks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brooms/bristle-material~titanium-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89955K388&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289955K388%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=240h-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2700 pages (at 45 pages/min), scraped 85 items (at 3 items/min) 2025-11-01 22:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/thread-size~m12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-halide-light-fixtures/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~gya108rrb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=240l-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caps/pipe-size~1-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a-frame-trucks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/helicoils/thread-size~m12/ 2025-11-01 22:56:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brooms/bristle-material~titanium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002090932%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cord-gaskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7397459384976&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7397459384976%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000074440%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/brooms/bristle-material~titanium/ 2025-11-01 22:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/compression-vacuum-tube-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-halide-light-fixtures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/270xl-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~mn1400/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compression-vacuum-tube-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chillers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~20-0000-mm/diameter~20-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/pin-type~for-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=270xl-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-128&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-021/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969505%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~mn1400&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001518725%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~mr-bat/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2744 pages (at 44 pages/min), scraped 86 items (at 1 items/min) 2025-11-01 22:57:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~20mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/abrasive-disc-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002096986%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-128&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 22:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drums/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-021&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/dowel-pins/diameter~20mm/ 2025-11-01 22:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009010%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/size~1-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71455K54&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271455K54%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~mr-bat&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cord-grips/enclosure-rating~ip66/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/pin-type~taper/material~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507081%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrasive-disc-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drums/material~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5560K59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225560K59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drums/material~brass/ 2025-11-01 22:57:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~26-03-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grips/environmental-rating~ip66&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-038/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/size~1-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/pin-type~taper/material~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cord-grips/environmental-rating~ip66/ 2025-11-01 22:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~10-000-mm/id~10-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5560K59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225560K59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969083%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~5-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~26-03mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-038&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000064886%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2787 pages (at 43 pages/min), scraped 87 items (at 1 items/min) 2025-11-01 22:58:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~26-03mm/ 2025-11-01 22:58:11 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:58:11 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 22:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~10mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504865%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5560K59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225560K59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~5-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/cable-awg~16-25/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~10mm/ 2025-11-01 22:58:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~11-3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:25 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129226%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chokers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/balls/diameter~1-2-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504865%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:32 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/cable-awg~16-25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~nic0078/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~11-300mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-055/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129226%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-161&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~r1810-2z/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561478%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balls/diameter~1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~11-300mm/ 2025-11-01 22:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504865%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000075802%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8082K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228082K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~nic0078&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/balls/diameter~1-2/ 2025-11-01 22:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~11-9997-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:58:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-055&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129226%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A232&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A232%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~r1810-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-machine-component-knobs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2836 pages (at 49 pages/min), scraped 88 items (at 1 items/min) 2025-11-01 22:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001495604%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002142991%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bspt-to-npt-pipe-fitting-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~11-9997mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2267N67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222267N67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/abrasive-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K645&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K645%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 22:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A232&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A232%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-machine-component-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~11-9997mm/ 2025-11-01 22:59:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/balls/diameter~1-4-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 22:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crown-locknuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bspt-to-npt-pipe-fitting-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrasive-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~nic0811/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A232&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A232%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balls/diameter~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/cable-awg~18-42/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cinching-straps/material~polyester-fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/drive-size~1-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crown-locknuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cord-grips/knockout-trade-size~pg-7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/balls/diameter~1-4/ 2025-11-01 22:59:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7700851996814&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cinching%20straps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7700851996814%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512635%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 22:59:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2653472997099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20drivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2653472997099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489923%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~nic0811&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:02 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:00:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=circle-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/cable-awg~18-42&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2883 pages (at 47 pages/min), scraped 90 items (at 2 items/min) 2025-11-01 23:00:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~nic1554/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~r3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089233%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grips/for-knockout-trade-size~pg-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cord-grips/material~bronze-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561567%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~nm715/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-109/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cord-grips/for-knockout-trade-size~pg-7/ 2025-11-01 23:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buckles/buckle-type~squeeze-release/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2267N47&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222267N47%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5106N26&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225106N26%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~nic1554&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~r3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/abs-round-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shim-stock-sheets/material~stainless-steel-2/material~301-stainless-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101408%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~nm715&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-109&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081492%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckles/buckle-type~squeeze-release&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-118/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grips/material~bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002106542%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505249%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abs-round-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9988424212043&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9988424212043%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562125%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=304-stainless-steel-shim-stock-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cord-grips/material~bronze/ 2025-11-01 23:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101408%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6762K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226762K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~9-16-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-118&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2928 pages (at 45 pages/min), scraped 92 items (at 2 items/min) 2025-11-01 23:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505249%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-sheets/thickness~0-012/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acetal-linear-ball-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~b47/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498207%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=circuit-breakers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101408%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~m11/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~9-16-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K646&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K646%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505249%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/thickness~0-012&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-linear-ball-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334596%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=b47-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~15/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~m11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001456969%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~pmnn4497/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~16-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454214%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:48 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:01:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rj-45-jack-locks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/316-stainless-steel-shim-stock-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chrome-moly-pipe/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~15-ga&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rj-45-plug-locks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/curb-markers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~pmnn4497&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~15-ga/ 2025-11-01 23:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~16-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 2970 pages (at 42 pages/min), scraped 94 items (at 2 items/min) 2025-11-01 23:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rj-45-jack-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=316-stainless-steel-shim-stock-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chrome-moly-pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chrome/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002085469%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~16-5-mm/ 2025-11-01 23:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rj-45-plug-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=curb-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1467T44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221467T44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/balls/diameter~2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct/connects-to~hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chrome&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~m35/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bottle-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=circulating-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~16-01-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001499846%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1467T44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221467T44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balls/diameter~2mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K411&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K411%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/connection-type~hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~m35&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-sheets/thickness~1-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bottle-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/balls/diameter~2mm/ 2025-11-01 23:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~16-025-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~16-01mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/duct/connection-type~hose/ 2025-11-01 23:03:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448469%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1467T44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221467T44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chuck-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3014 pages (at 44 pages/min), scraped 95 items (at 1 items/min) 2025-11-01 23:03:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~16-01mm/ 2025-11-01 23:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/thickness~1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-209/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~16-025mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamp-feet&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~16-3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/thread-size~2-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001461876%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chuck-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~16-025mm/ 2025-11-01 23:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~sp393/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/thread-size~3-8-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334446%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-209&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sheets/material~aluminum-2/thickness~2-000-mm/thickness~2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/curtains/color~white/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~16-3mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6648N49&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226648N49%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-249/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001501649%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~sp393&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~16-3mm/ 2025-11-01 23:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct/for-use-with~beverage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K741&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K741%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/thickness~2mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=curtains/color~white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~19-076-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079455%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-sheets/thickness~7-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carbon-steel-rectangular-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-249&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/aluminum-sheets/thickness~2mm 2025-11-01 23:04:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8163625184607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8163625184607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3061 pages (at 47 pages/min), scraped 97 items (at 2 items/min) 2025-11-01 23:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7604K47&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227604K47%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/for-use-with~beverage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/digital-bench-scales/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/380mxl-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001494436%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~19-076mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/thickness~7-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967788%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-steel-rectangular-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~389/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89785K391&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289785K391%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~19-076mm/ 2025-11-01 23:04:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/robotics/robot-cell-component~vacuum-lifter/for-robot-arm-manufacturer~doosan/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001462926%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9394K101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229394K101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=digital-bench-scales&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=380mxl-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-style~hex-with-pilot-recess/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9057K89&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229057K89%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/circlip-pliers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/voltage~12v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=389-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~vacuum-lifter/manufacturer~doosan&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carbon-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~4-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-spray-containers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/tip-style~pilot&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9985525866263&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9985525866263%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002096768%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cardboard-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=circlip-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/voltage~12v&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/pressure-class~1000/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2653472997099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20drivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2653472997099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002238943%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3103 pages (at 42 pages/min), scraped 101 items (at 4 items/min) 2025-11-01 23:05:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~4-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-spray-containers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/batteries/voltage~12v/ 2025-11-01 23:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cardboard-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000132352%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesive-back-vertical-scales/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=class-1000-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/custom-shims/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/voltage~36v&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct-hose-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~42-04-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-keys/finish~nickel-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/batteries/voltage~36v/ 2025-11-01 23:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000132352%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-back-vertical-scales&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rod-bumpers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342665684929&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342665684929%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001519486%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=custom-shims&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-hose-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~42-04mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pintle-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000132352%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~3-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~44-025-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-bumpers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~42-04mm/ 2025-11-01 23:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cargo-load-stabilizers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3148 pages (at 45 pages/min), scraped 101 items (at 0 items/min) 2025-11-01 23:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cut-to-length-springs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-nuts/fastener-strength-grade-class~astm-grade-2h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-68/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~3-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~44-025mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cargo-load-stabilizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~3-05-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~46-051-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8523081672262&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8523081672262%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001570509%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~3-5-8/ 2025-11-01 23:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cut-to-length-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~44-025mm/ 2025-11-01 23:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bottles/capacity~6-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539762%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:39 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-nuts/fastener-strength-grade-class~astm-grade-2h&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 23:06:40 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:06:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cleaned-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acetal-sheet-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A257&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A257%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~46-051mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bulkheads/fitting-type~connector-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506107%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drawer-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bottles/capacity-weight-~6-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:06:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~46-051mm/ 2025-11-01 23:06:58 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:06:58 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~5-01-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bottles/capacity-weight-~6-gal-/ 2025-11-01 23:07:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acetal-strips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3194 pages (at 46 pages/min), scraped 102 items (at 1 items/min) 2025-11-01 23:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulkheads/type~connector&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506107%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drawer-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ductile-iron-rod-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-bender-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bulkheads/type~connector/ 2025-11-01 23:07:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drawer-slide-rails/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~5-01mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thumb-nuts/material~acetal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506107%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~5-01mm/ 2025-11-01 23:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~3-17-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ductile-iron-rod-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-tube-tees/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drawer-slide-rails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-bladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~39-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-thumb-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:07:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/battery-operated-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-vacuum-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~3-17-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-tube-tees&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/material~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~20-042-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~39-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/3d-printers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000062388%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~3-8-2/diameter~0-375&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3238 pages (at 44 pages/min), scraped 102 items (at 0 items/min) 2025-11-01 23:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-operated-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~39-64/ 2025-11-01 23:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-vacuum-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-trash-pails/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000103678%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000108941%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~20-042mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/diameter~3-8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cable/material~iron/ 2025-11-01 23:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~1-259/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-wedges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~20-042mm/ 2025-11-01 23:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~22-04-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-trash-pails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~3-8mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/number-of-wires~41/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/roller-chain-channels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~1-259&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-wedges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-shank-grinding-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire/diameter~3-8mm/ 2025-11-01 23:08:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~22-04mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~23-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506243%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/american-standard-toilet-gaskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:08:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~22-04mm/ 2025-11-01 23:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/number-of-wires~41&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roller-chain-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89965K414&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289965K414%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~1-561/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3281 pages (at 43 pages/min), scraped 102 items (at 0 items/min) 2025-11-01 23:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548123%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conical-seals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~23-000mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clamps/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~25-01-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=american-standard-toilet-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/beads/comparable-to~lexan/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9936K45&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229936K45%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~23-000mm/ 2025-11-01 23:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acme-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~1-561&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dvi-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conical-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamps/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~25-01mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001501558%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=beads/comparable-to~lexan&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/an-961-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acme-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/clamps/material~aluminum/ 2025-11-01 23:09:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~25-01mm/ 2025-11-01 23:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8950K931&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228950K931%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hi-lok-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9970513714292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9970513714292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313126%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dvi-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bumpers/color~yellow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clamps/material~zinc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~5-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9363K31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229363K31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=an-961-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/garolite-sheets/material-grade~garolite-le-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/an-970-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3322 pages (at 41 pages/min), scraped 105 items (at 3 items/min) 2025-11-01 23:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ball-checks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/box-sealers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bumpers/color~yellow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamps/material~zinc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/beads/comparable-to~zytel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~5-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7150684641693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7150684641693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9791062983264&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9791062983264%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512333%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=e-glass-grade-garolite-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=an-970-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-checks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000152578%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=box-sealers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~29-500mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=beads/comparable-to~zytel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~29-500mm/ 2025-11-01 23:10:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/flute-type~spiral-2/flute-type~open/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metals/fabrication~heat-treated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~18-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/adhesive-type~concrete-anchoring/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~59-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9970513714292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9970513714292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310071%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metals/shape~half-round/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-curtain-fans/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bumpers/mount-type~hook-and-loop&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9613K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229613K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/bit-style~open-spiral-flute&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metals/fabrication~heat-treated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~18-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7150684641693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7150684641693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003011009%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchoring-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3367 pages (at 45 pages/min), scraped 106 items (at 1 items/min) 2025-11-01 23:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~59-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metals/shape~half-round&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001926883%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1062337176277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1062337176277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001525387%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001492944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-curtain-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~59-64/ 2025-11-01 23:11:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9613K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229613K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1062337176277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1062337176277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002308998%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/class-c-fire-extinguishers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bumpers/thread-size~1-1-4-7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~6-8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1289T18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221289T18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/easy-access-pipe-flanges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~2-6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~3l680/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9613K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229613K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/class-d-fire-extinguishers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=class-c-fire-extinguishers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bumpers/thread-size~1-1-4-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~6-8-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1289T18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221289T18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=easy-access-pipe-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~2-6-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3l680-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7150684641693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7150684641693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:11:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~6-8-mm/ 2025-11-01 23:11:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=class-d-fire-extinguishers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001494572%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3411 pages (at 44 pages/min), scraped 106 items (at 0 items/min) 2025-11-01 23:12:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-driven-piston-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~9-025-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:12:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1289T18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221289T18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brackets/bracket-type~connecting/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bumpers/thread-size~3-4-10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-exhaust-ball-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5195T51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225195T51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brackets/bracket-type~triple-90-angle/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-driven-piston-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~9-025mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-pressure-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/bracket-type~connecting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bumpers/thread-size~3-4-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buckets/capacity~24-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-exhaust-ball-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~9-025mm/ 2025-11-01 23:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/disc-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/bracket-type~triple-90-angle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561267%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7150684641693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7150684641693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334890%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearing-extractors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/edge-grommets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562062%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3626T42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223626T42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity-weight-~24-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-cement/for-use-on~cpvc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:12:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=disc-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=33125T483&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2233125T483%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7150684641693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7150684641693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334890%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/buckets/capacity-weight-~24-gal-/ 2025-11-01 23:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-extractors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=edge-grommets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6814180174308&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20cement&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6814180174308%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930253%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3456 pages (at 45 pages/min), scraped 107 items (at 1 items/min) 2025-11-01 23:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/bearings/oil-embedded-flanged-sleeve-bearings-7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acrylic/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3626T42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223626T42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/edge-protectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7150684641693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7150684641693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334890%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose-festoon-systems/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearing-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose-lockouts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/plastic/clear-scratch-and-uv-resistant-acrylic-tubes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acrylic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=edge-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/bearings/oil-embedded-flanged-sleeve-bearings-7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%223334397114803%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:13:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3626T42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223626T42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ball-joint-rod-ends/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose-festoon-systems&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose-lockouts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8532K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228532K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/for-use-with~neon/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/edging/material~bronze-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clean-room-chairs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:59 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/rod-ends-and-linkages/ball-joint-rod-ends-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:13:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/performance~metal-detectable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/methylated-spirits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-acme-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/casters/wheel-diameter~3-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dispensing-containers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3496 pages (at 40 pages/min), scraped 109 items (at 2 items/min) 2025-11-01 23:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~neon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=edging/material~bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clean-room-chairs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~5-700-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=methylated-spirits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965301%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/metal-detectable-cable-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-acme-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/edging/material~bronze/ 2025-11-01 23:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=casters/wheel-diameter~3-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dispensing-containers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/anchors/metal-detectable-cable-ties/ 2025-11-01 23:14:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/american-wire-gauge-awg~8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bundling-straps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1981438481801&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1981438481801%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001922726%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~5-7-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coatings/for-use-on~plastic-1/for-use-on~polyimide/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-particle-sensors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brackets/screw-size~m3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2320T41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222320T41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506394%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/cable-designation~uf-b/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~4-4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/wire-gauge~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bundling-straps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5195T53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225195T53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~polyimide-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-particle-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/for-screw-size~m3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cable/wire-gauge~8/ 2025-11-01 23:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~9-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/coatings/for-use-on~polyimide-plastic/ 2025-11-01 23:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~4-4-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3541 pages (at 45 pages/min), scraped 111 items (at 2 items/min) 2025-11-01 23:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/industry-designation~uf-b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/brackets/for-screw-size~m3/ 2025-11-01 23:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buckets/capacity~9-1-2-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-beveled-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brackets/screw-size~no-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506143%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bungs/thread-type~ngo/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire/industry-designation~uf-b/ 2025-11-01 23:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/shaft-diameter~2-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97060A114&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297060A114%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-conduit/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coiled-air-lines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity-weight-~9-1-2-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~9-16/ 2025-11-01 23:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-beveled-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/for-screw-size~no-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bungs/thread-type~ngo&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-shaft-diameter~2-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/buckets/capacity-weight-~9-1-2-gal-/ 2025-11-01 23:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elastic-mesh-sleeves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/brackets/for-screw-size~no-8/ 2025-11-01 23:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rotary-rasps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-conduit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9904654906013&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9904654906013%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138997%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/for-shaft-diameter~2-7-8/ 2025-11-01 23:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/catch-pans/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rotary-speed-sensors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~1204k/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elastic-mesh-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coiled-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cleaners/for-use-on~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-rasps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9904654906013&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9904654906013%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138997%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3584 pages (at 43 pages/min), scraped 112 items (at 1 items/min) 2025-11-01 23:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-temperature-continuous-flex-cable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=catch-pans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-speed-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~1204k&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/counterbores/screw-size~m4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081845%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~48-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-entry-grips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cleaners/for-use-on~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-cable-ties-and-mounts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9904654906013&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9904654906013%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138997%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bar-magnets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cleaners/for-use-on~iron/ 2025-11-01 23:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-temperature-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=counterbores/for-screw-size~4-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/number-of-wires~10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~48-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-entry-grips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:47 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:16:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/counterbores/for-screw-size~4-mm/ 2025-11-01 23:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bar-racks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/metal-cable-ties-and-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bar-magnets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507334%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rotary-vane-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/distribution-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:16:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/brackets/metal-cable-ties-and-mounts/ 2025-11-01 23:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/number-of-wires~10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tie-down-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/outer-jacket-color~gray/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3627 pages (at 43 pages/min), scraped 112 items (at 0 items/min) 2025-11-01 23:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bar-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-powered-winch-tuggers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001557045%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/diverting-ball-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507334%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-vane-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=distribution-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K79&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K79%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/tie-down-rings-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/tube-od~2-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/outer-insulation-color~gray&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-powered-winch-tuggers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diverting-ball-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/anchors/tie-down-rings-6/ 2025-11-01 23:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~16002-2z/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dividers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507334%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire/outer-insulation-color~gray/ 2025-11-01 23:17:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cylinder-tags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:40 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/shaft-diameter~42-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brackets/mounting-location~loop-hanger/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/for-tube-od~2-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/counters/measurement-unit~inches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~16002-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dividers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:17:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/adapters/for-tube-od~2-1-2/ 2025-11-01 23:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylinder-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-shaft-diameter~42mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002097044%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/length~25-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-temperature-terminal-and-splice-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3670 pages (at 43 pages/min), scraped 113 items (at 1 items/min) 2025-11-01 23:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/mounting-location~loop-hanger&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/tube-od~3-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=counters/unit-of-measure~inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/for-shaft-diameter~42mm/ 2025-11-01 23:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bench-centers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002380319%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/counters/unit-of-measure~inches/ 2025-11-01 23:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-glands/thread-size~pg-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/length~28-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-cord-grips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/elbow-type~with-mounting-tab/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~25-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9164T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229164T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/for-tube-od~3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~3v315/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bench-centers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001604169%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/adapters/for-tube-od~3-16/ 2025-11-01 23:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-glands/thread-size~pg-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~28-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-cord-grips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~3v335/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/elbow-style~with-mounting-tab&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2333N136&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222333N136%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6782425493977&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6782425493977%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454988%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/air-hose/length~28-ft-/ 2025-11-01 23:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/round-over-router-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/round-reflectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3v315-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:18:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/elbows/elbow-style~with-mounting-tab/ 2025-11-01 23:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=69915K118&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2269915K118%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/for-use-with~ammonia/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3712 pages (at 42 pages/min), scraped 114 items (at 1 items/min) 2025-11-01 23:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3v335-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/countersinks/screw-size~5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2333N136&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222333N136%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-over-router-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-reflectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rpm-indicators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-vacuum-chambers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=69915K118&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2269915K118%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/braided-stainless-steel-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/for-use-with~ammonia&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/for-screw-size~5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2333N136&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222333N136%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/d-rings/mount-type~adhesive/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rpm-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965242%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=69915K118&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2269915K118%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/countersinks/for-screw-size~5-16/ 2025-11-01 23:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=colored-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=braided-stainless-steel-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=colossus-casters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9066N441&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229066N441%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/for-use-with~calcium-chloride/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:19:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~0-096/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=d-rings/mount-type~adhesive&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:00 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:20:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/length~96/length~8-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alcohol/concentration~91/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1803157841548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1803157841548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010482%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/d-shape-knockout-punches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3755 pages (at 43 pages/min), scraped 115 items (at 1 items/min) 2025-11-01 23:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/for-use-with~calcium-chloride&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clevis-pins/diameter~1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-096&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fitting-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~8-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alcohol/concentration~91-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000125973%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=d-shape-knockout-punches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/air-hose/length~8-ft-/ 2025-11-01 23:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clevis-pins/diameter~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/alcohol/concentration~91-/ 2025-11-01 23:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fitting-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~1635-2rs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~1635-2z/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/specifications-met~nas1564j16-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose-carriers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cement/for-use-on~cpvc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1255618309250&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1255618309250%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000128883%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/beveled-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~1635-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~1635-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cement/for-use-on~rubber-2/for-use-on~epdm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/specifications-met~nas1564j16-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose-carriers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~0-1695/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cement/for-use-on~cpvc-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002106351%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3798 pages (at 43 pages/min), scraped 115 items (at 0 items/min) 2025-11-01 23:21:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002103202%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089756%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=beveled-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cement/for-use-on~cpvc-plastic/ 2025-11-01 23:21:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bevelers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cement/for-use-on~epdm-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-inlets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barrier-style-terminal-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~3vx800/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5482K164&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225482K164%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cement/for-use-on~epdm-rubber/ 2025-11-01 23:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass/shape~round-tube/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-1695&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-limit-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bevelers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits-for-wood-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-inlets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barrier-style-terminal-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dolly-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3vx800-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dome-mirrors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000162645%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass/shape~round-tube&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-limit-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-(schedule-40)-unthreaded-pipe/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5551584616364&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5551584616364%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002311201%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-allen-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dolly-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dome-mirrors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/roller-chain/chain-standard~ansi/roller-chain-trade-number~80-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3842 pages (at 44 pages/min), scraped 116 items (at 1 items/min) 2025-11-01 23:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/thread-size~1-2-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/fitting-type~trap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-(schedule-40)-unthreaded-pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002464592%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~2208-2rs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/420xl-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-allen-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ansi-80-2-roller-chain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~1-2-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/40mxl-timing-belts already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1906N31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221906N31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dampers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-chuck-bodies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455324%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~2208-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/40mxl-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=420xl-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-alloy-360-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/door-closers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-oil-seals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002115764%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/450h-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dampers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455324%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=40mxl-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-alloy-360-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/fitting-type~union-1/for-use-with~hydraulic-fluid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-closers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/base-mounted-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-oil-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3887 pages (at 45 pages/min), scraped 117 items (at 1 items/min) 2025-11-01 23:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=450h-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clips/clamp-type~latch/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455324%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~4-7-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~460j8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lid-supports/environment~clean-room/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/fitting-type~union-1/for-use-with~hydraulic-fluid-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=base-mounted-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~0-368/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clips/clamp-type~latch&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/overall-length~1-3-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001919079%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:35 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~4-7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=460j8-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9975836744491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9975836744491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001927545%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/clean-room-lid-supports&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rumble-strips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bushings/for-shaft-diameter~4-7-16/ 2025-11-01 23:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-368&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/lubrication~lubricated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/pipe-size~4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5071A101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225071A101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hinges/clean-room-lid-supports/ 2025-11-01 23:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~1-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~22313/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clips/finish~oil-coated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000010963%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rumble-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001606396%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3929 pages (at 42 pages/min), scraped 117 items (at 0 items/min) 2025-11-01 23:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/door-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/lubrication~lubricated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5071A101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225071A101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/fitting-type~union-1/material~stainless-steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:14 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~1-3-8&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 23:24:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~22313&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clips/finish~oil-coated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/pipe-size~4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001973327%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/door-levers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001919079%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107968%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-1695/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9975836744491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9975836744491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001531334%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129622%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5071A101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225071A101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95524A134&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295524A134%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~0-234/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/pem-part-number~cls-440-3/pem-part-number~s-440-3zi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-levers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-1695&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ceramic-coated-linear-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~2307-2rs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/material~iron-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504439%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-234&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hinges/material~kevlar-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=s-440-3-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000147754%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ceramic-coated-linear-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~2307-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 3975 pages (at 46 pages/min), scraped 118 items (at 1 items/min) 2025-11-01 23:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-and-cable-crimpers/for-wire-gauge~28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000014217%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504439%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002084425%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5500541266190&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20and%20cable%20crimpers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5500541266190%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001992196%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~4l230/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/material~kevlar&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-british-standard-thread-pipe-nipples/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/length~15-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hinges/material~kevlar/ 2025-11-01 23:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-and-cable-crimpers/for-wire-gauge~3-0&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/battery-connection~quick-disconnect-terminal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504439%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5500541266190&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20and%20cable%20crimpers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5500541266190%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001992196%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=4l230-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-slotted-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5500541266190&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20and%20cable%20crimpers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5500541266190%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001999835%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-british-standard-thread-pipe-nipples&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-operated-cleaning-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~15-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/battery-connection-type~quick-disconnect&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001610323%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5500541266190&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20and%20cable%20crimpers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5500541266190%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001992196%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-jobsite-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~3002/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-slotted-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/batteries/battery-connection-type~quick-disconnect/ 2025-11-01 23:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hinges/mounting-location~edge/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/battery-connection~alligator-clip/battery-connection-protection~sheathed/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4021 pages (at 46 pages/min), scraped 118 items (at 0 items/min) 2025-11-01 23:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~made-to-order/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/shaft-diameter~2-7-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~3002&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-jobsite-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/mounting-style~edge&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/battery-connection-type~sheathed-alligator-clip&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000025343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-point-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-oxide-sanding-sponges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143724%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hinges/mounting-style~edge/ 2025-11-01 23:26:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/batteries/battery-connection-type~sheathed-alligator-clip/ 2025-11-01 23:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/safety-barrels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-shaft-diameter~2-7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ceramic-paint/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~made-to-order&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-and-cable-crimpers/military-specification~mil-c-22520-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/battery-size~1-2-aa/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-oxide-sanding-sponges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/couplings/for-shaft-diameter~2-7-16/ 2025-11-01 23:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/thread-size~m3-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bushings/for-shaft-diameter~made-to-order/ 2025-11-01 23:26:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-barrels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/battery-connection-type~wire-leads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/battery-size~16340/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-tool-oilers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ceramic-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-and-cable-crimpers/military-specifications-met~mil-c-22520-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/battery-size~1-2-aa&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/batteries/battery-connection-type~wire-leads/ 2025-11-01 23:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4065 pages (at 44 pages/min), scraped 118 items (at 0 items/min) 2025-11-01 23:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/thread-size~m56/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~m3-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:27:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire-and-cable-crimpers/military-specifications-met~mil-c-22520-1/ 2025-11-01 23:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/appliance-cords/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003030652%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-2344/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000028439%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/battery-size~16340&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~11-2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-stainless-steel-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-tool-oilers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2164N16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222164N16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~m56&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003030829%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=appliance-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-2344&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448923%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1109K551&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221109K551%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~11-2mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/shaft-diameter~7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-stainless-steel-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~4l560/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507667%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/shank-size~11-2mm/ 2025-11-01 23:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=96511A853&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2296511A853%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/black-oxide-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alcohol-dispensers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/shaft-diameter~9-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-shaft-diameter~7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=4l560-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-and-cable-strippers/for-wire-gauge~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4109 pages (at 44 pages/min), scraped 120 items (at 2 items/min) 2025-11-01 23:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-supports&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/couplings/for-shaft-diameter~7-8/ 2025-11-01 23:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=96511A853&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2296511A853%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=506136630644&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20and%20cable%20strippers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B506136630644%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001989984%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=black-oxide-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alcohol-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/black-oxide-steel-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-shaft-diameter~9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-grip-lock-hose-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8079K101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228079K101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer~k2-energy/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/couplings/for-shaft-diameter~9-16/ 2025-11-01 23:28:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~19-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/closed-style-linear-ball-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=96511A853&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2296511A853%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/deformed-thread-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=black-oxide-steel-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026147%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-grip-lock-hose-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-261/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer~k2-energy&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compression-force-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1796A13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221796A13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=closed-style-linear-ball-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:56 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:28:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=deformed-thread-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:28:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/thread-type~npsm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002260702%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alignment-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-261&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4156 pages (at 47 pages/min), scraped 122 items (at 2 items/min) 2025-11-01 23:29:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:29:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2803K9&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222803K9%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/demolition-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~2-3-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cga-520-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/for-use-with~ammonia/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504205%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-302/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-type~npsm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000103199%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alignment-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=96511A936&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2296511A936%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000151504%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~0-69/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=demolition-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2434K519&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222434K519%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cga-520-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~anhydrous-ammonia&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-302&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/couplings/for-use-with~anhydrous-ammonia/ 2025-11-01 23:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-hose-elbows/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/length~4-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505839%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/arbors/for-use-with~handheld-disc-sanders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-jam-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~2-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-69&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compression-spring-retainers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/thread-size~2-1-2-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98042A011&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298042A011%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000093475%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-hose-elbows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~4-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4200 pages (at 44 pages/min), scraped 125 items (at 3 items/min) 2025-11-01 23:30:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=arbors/for-use-with~hand-held-disc-sanders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-jam-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~2-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~2-1-2-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001606923%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98042A011&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298042A011%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/arbors/for-use-with~hand-held-disc-sanders/ 2025-11-01 23:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-thread-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000149613%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454244%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/desk-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesive-back-foil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/blades/blade-type~general-purpose-straight-cutting/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~3-3-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:32 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~2-1-4&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 23:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98042A011&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298042A011%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6698K26&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226698K26%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-thread-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/blades/precision-knife-blade-type~straight-cutting/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/finish~zinc-plated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000080165%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=desk-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~liquid-helium-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-back-foil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2440K47&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222440K47%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969101%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-manifolds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~0-8755/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blades/blade-type~general-purpose-straight-cutting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/finish~zinc-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hinges/overall-width~32-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3441N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223441N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4244 pages (at 44 pages/min), scraped 127 items (at 2 items/min) 2025-11-01 23:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~0-8764/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/blades/blade-type~general-purpose-straight-cutting/ 2025-11-01 23:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8304400284430&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8304400284430%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-manifolds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/number-of-pieces~117/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-8755&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compression-tube-straights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~32mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-brushes/handle-material~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/length~6-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000135319%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-8764&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hinges/overall-width~32mm/ 2025-11-01 23:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~20-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesive-back-sandpaper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/number-of-pieces~117&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000162042%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-brushes/handle-material~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~6-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nelson-anchors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001540611%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~4-15-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~20mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~mild-acids&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire-brushes/handle-material~fiberglass/ 2025-11-01 23:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-back-sandpaper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001627407%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000148004%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:31:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/shank-size~20mm/ 2025-11-01 23:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179301%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/weld-on-stud-anchors-for-wet-concrete&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4290 pages (at 46 pages/min), scraped 128 items (at 1 items/min) 2025-11-01 23:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/number-of-pieces~29/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~0-947/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1530A71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221530A71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~4-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clss-024-2-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-boxes/cable-designation~sjeoow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesive-cartridges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/anchors/weld-on-stud-anchors-for-wet-concrete/ 2025-11-01 23:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000134092%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/pem-part-number~clss-032-0/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/number-of-pieces~29&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-947&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/arm-knobs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-boxes/cable-designation~sjoow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12575A621&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212575A621%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-boxes/industry-designation~sjeoow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-cartridges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clss-024-2-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000084199%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179301%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/electrical-boxes/industry-designation~sjeoow/ 2025-11-01 23:32:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clss-032-0-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=arm-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-boxes/industry-designation~sjoow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-39/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-boxes/cable-designation~stow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/number-of-pieces~7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/angle-identifiers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:32:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/electrical-boxes/industry-designation~sjoow/ 2025-11-01 23:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-weld-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179301%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4332 pages (at 42 pages/min), scraped 130 items (at 2 items/min) 2025-11-01 23:33:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-39&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-boxes/industry-designation~stow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/number-of-pieces~7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alumina-oxide-beads/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=angle-identifiers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/roller-chain-trade-number~240/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/oversized-wear-resistant-d2-tool-steel-rods-and-discs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001494746%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-weld-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:19 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/electrical-boxes/industry-designation~stow/ 2025-11-01 23:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001555292%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%227798516857014%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9473N348&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229473N348%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~5l370/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=computer-monitor-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alumina-oxide-beads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/roller-chain-trade-size~240&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-plated-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89945K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289945K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/angle-seat-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain/roller-chain-trade-size~240/ 2025-11-01 23:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5l370-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=concealed-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/for-use-on~asphalt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~2587678-8005/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-cable-splices/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/roller-chain-trade-number~60-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-plated-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=angle-seat-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4378 pages (at 46 pages/min), scraped 132 items (at 2 items/min) 2025-11-01 23:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mica-gaskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~4-15-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~asphalt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~2587678-8005&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable-splices&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dow-corning-molykote-3452-grease/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/roller-chain-trade-size~60-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/for-use-on~cast-iron/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099833%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coach-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain/roller-chain-trade-size~60-4/ 2025-11-01 23:34:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mica-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~4-15mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2286N29&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222286N29%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~1-207/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dial-indicator-stands/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dow-corning-molykote-3452-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~cast-iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/shank-size~4-15mm/ 2025-11-01 23:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coach-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/for-use-with~neon/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930023%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/blind-rivet-installation-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-207&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/for-use-on~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicator-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~1-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~49-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-coil-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000134355%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~neon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4420 pages (at 42 pages/min), scraped 133 items (at 1 items/min) 2025-11-01 23:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-cord-ends/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blind-rivet-installation-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7197N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227197N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~1-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965301%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~49-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/adhesives/for-use-on~fiberglass/ 2025-11-01 23:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum/material~aluminum-2/material~6061-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-connectors/application~audio&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~1-5-8/ 2025-11-01 23:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cord-ends&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coating-corrosion-inhibitors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum/material~aluminum-2/material~6101-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/micro-drill-attachments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000114715%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ansi-flared-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1712501638756&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1712501638756%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002344780%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~1-3-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum/material~aluminum/material~6061-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1860N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221860N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coating-corrosion-inhibitors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~65-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum/material~aluminum/material~6101-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~1-11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=micro-drill-attachments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512136%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ansi-flared-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512093%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/for-use-on~plastic-1/for-use-on~polycarbonate/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sanitary-napkin-receptacle-liners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4463 pages (at 43 pages/min), scraped 135 items (at 2 items/min) 2025-11-01 23:36:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~65-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9397T12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229397T12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000097824%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~1-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/roller-chain-trade-number~35/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/drive-size~1-11-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/containers/capacity-weight-~65-gal-/ 2025-11-01 23:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sanitary-stairs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:36:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/drive-size~1-11-16-1/ 2025-11-01 23:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~polycarbonate-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/thread-size~m27/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-napkin-receptacle-liners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~6201-2rs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/thread-size~m3-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/drive-size~1-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coatings/for-use-on~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/trade-size~35&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/adhesives/for-use-on~polycarbonate-plastic/ 2025-11-01 23:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-stairs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/drive-size~1-16-1/ 2025-11-01 23:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain/trade-size~35/ 2025-11-01 23:36:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-connectors/electrical-connection~vehicle/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~m27&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6201-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/6-chain/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wipes/material~microfiber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~m3-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/anti-glare-film/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333390%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143447%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~22-0-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000028439%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/coatings/for-use-on~fiberglass/ 2025-11-01 23:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-connectors/connection-type~vehicle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/trade-size~6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4507 pages (at 44 pages/min), scraped 136 items (at 1 items/min) 2025-11-01 23:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=microfiber-wipes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire-connectors/connection-type~vehicle/ 2025-11-01 23:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anti-glare-film&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dial-indicators/manufacturer-model-number~25-441j/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain/trade-size~6/ 2025-11-01 23:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~22-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coatings/for-use-on~leather-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~7-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/drive-size~100-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~22-mm/ 2025-11-01 23:37:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/anti-tarnish-bags already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~6205-2z/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~4ph55/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/calipers/manufacturer-model-number~26-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~25-441j&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8659884989431&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drive%20sockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8659884989431%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489960%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/antifog-lens-cleaning-wipes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~leather&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002086190%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99940A320&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299940A320%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6205-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/coatings/for-use-on~leather/ 2025-11-01 23:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~4ph55&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000132132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/manufacturer-model-number~26-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/anti-tarnish-bags already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002137893%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=antifog-lens-cleaning-wipes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002111021%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~6207-2rs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8826536049195&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8826536049195%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002091299%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99940A320&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299940A320%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/material~polypropylene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2267N82&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222267N82%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4549 pages (at 42 pages/min), scraped 136 items (at 0 items/min) 2025-11-01 23:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~90-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1962A41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221962A41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-alloy-6013/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-0328/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509869%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:13 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:38:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6207-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99940A320&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299940A320%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/thread-size~m90/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~90-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101675%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1962A41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221962A41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-alloy-6013&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-0328&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dial-indicators/manufacturer-model-number~513-405-10e/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/containers/capacity-weight-~90-ml/ 2025-11-01 23:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hitch-pin-locks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001525146%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:40 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-splitters/electrical-connection~coaxial/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~m90&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dial-indicators/manufacturer-model-number~513-446-16a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-links/roller-chain-trade-number~06b/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1962A41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221962A41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A150&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A150%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~513-405-10e&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454649%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~9-4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138432%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coaxial-cable-splitters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hitch-pin-locks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~513-446-16a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-links/roller-chain-trade-size~06b-ss&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4594 pages (at 45 pages/min), scraped 138 items (at 2 items/min) 2025-11-01 23:39:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002137423%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain-links/roller-chain-trade-size~06b-ss/ 2025-11-01 23:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/formulation~polymer/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/canisters/for-use-with~butane-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~9-4mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-links/roller-chain-trade-number~100/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-connectors/terminal-material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:21 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~1-85/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/shank-size~9-4mm/ 2025-11-01 23:39:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:39:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/pipe-size~12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/formulation~polymer&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=butane-canisters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/converters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-links/roller-chain-trade-size~100-ss&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coiled-cable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000136829%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-connectors/terminal-material~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002084956%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hoist-anchors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-85&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain-links/roller-chain-trade-size~100-ss/ 2025-11-01 23:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire-connectors/terminal-material~brass/ 2025-11-01 23:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6430A798&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226430A798%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brazed-plate-heat-exchangers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000057015%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/convex-radius-end-mills/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=converters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dial-indicators/manufacturer-model-number~711gcsz/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coiled-cable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mig-gun-insulators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/astm-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4634 pages (at 40 pages/min), scraped 139 items (at 1 items/min) 2025-11-01 23:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/shape~straight/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brazed-plate-heat-exchangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=convex-radius-end-mills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~5vx590/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~711gcsz&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rotozip-spiral-saw-cutoff-wheels-for-metal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mig-gun-insulators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=astm-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/drive-size~3-7-8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002119405%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/shape~straight&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dial-indicators/manufacturer-model-number~74-105997/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8659884989431&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drive%20sockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8659884989431%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489978%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5vx590-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saws/rotozip-spiral-saw-cutoff-wheels-for-metal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mig-welding-supplies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conveyor-belt-lacing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000124163%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99940A230&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299940A230%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/manufacturer~break-free/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=661583884380&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B661583884380%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%222310962390541%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~74-105997&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~10-023-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mig-welding-supplies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-belt-lacing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-outlets/outlet-box-component~box/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~5vx730/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-081/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4677 pages (at 43 pages/min), scraped 140 items (at 1 items/min) 2025-11-01 23:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~16-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=break-free-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3979A87&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223979A87%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~10-023mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-outlets/outlet-box-component~box&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~10-023mm/ 2025-11-01 23:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/thread-size~4-48/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-081&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~16-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/finish~bronze-plated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5vx730-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2359415142990&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2359415142990%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002364398%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-bronze-chisels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001530235%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-cutters/for-use-on~steel-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~16-mm/ 2025-11-01 23:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/thread-size~4-48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:41 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-081&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 23:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/5vx950-v-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:41:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/calipers/manufacturer-model-number~829a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/finish~bronze-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~2-9-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-bronze-chisels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334791%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~52-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001541884%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:57 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-cutters/for-use-on~steel-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:41:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:41:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5vx950-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/manufacturer-model-number~829a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/astm-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-links/roller-chain-trade-number~c2080h/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~2-9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4722 pages (at 45 pages/min), scraped 141 items (at 1 items/min) 2025-11-01 23:42:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/drive-size~52-mm-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/breakers/current~80-amp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-sockets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8826536049195&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8826536049195%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002113028%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~2-9-16/ 2025-11-01 23:42:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/drive-size~52-mm-1/ 2025-11-01 23:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-carts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:21 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2100A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222100A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:21 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:42:21 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:42:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=astm-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/astm-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-links/roller-chain-trade-size~c2080h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-chain/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=breakers/current~80a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mil-r-25988-o-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain-links/roller-chain-trade-size~c2080h/ 2025-11-01 23:42:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/breakers/current~80a/ 2025-11-01 23:42:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/scissor-sharpeners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=astm-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-chain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-tape/color~black-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mil-r-25988-o-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/fitting-type~reducer-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~br2032/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/specifications-met~astm-a576&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-ferrules&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scissor-sharpeners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/thread-size~2-56/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001931087%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~6805/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002085579%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4766 pages (at 44 pages/min), scraped 142 items (at 1 items/min) 2025-11-01 23:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-tape/color~black&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/calipers/distance-measured~0-to-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~br2032&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-1253/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-tape/color~red/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50925K118&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250925K118%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/electrical-tape/color~black/ 2025-11-01 23:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~2-56&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~90-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6805&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002091699%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~bx28/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000031867%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081033%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/maximum-measurement~24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7701K751&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227701K751%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-1253&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-tape/color~red&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/for-use-with~hydraulic-fluid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~90-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/calipers/maximum-measurement~24/ 2025-11-01 23:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550030%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=704271490290&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B704271490290%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522000%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bx28-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~90-mm/ 2025-11-01 23:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sharpening-stones/abrasive-material~diamond/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~140-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adjustable-nozzles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=76455A42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2276455A42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~hydraulic-fluid-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:43:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A463&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A463%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mil-spec-heat-transfer-paste/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/specifications-met~din-2353&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/connectors/for-use-with~hydraulic-fluid-1/ 2025-11-01 23:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4810 pages (at 44 pages/min), scraped 145 items (at 3 items/min) 2025-11-01 23:44:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diamond-sharpening-stones&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~140mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-nozzles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~15-0508-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paint/paint-type~chalkboard/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/military-specification~ms51575-6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mil-spec-heat-transfer-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~140mm/ 2025-11-01 23:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/thread-size~7-8-14/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~2-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cam-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~15-0508mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chalkboard-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/specifications-met~ms51575-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-extrusion-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~2-15-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~15-0508mm/ 2025-11-01 23:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~7-8-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bolts/specifications-met~ms51575-6/ 2025-11-01 23:44:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~2-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cam-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-wire-marking-stickers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/die-grinder-accessories/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:44:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~ax103/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-extrusion-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~2-15-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/combination-wrenches/size~2-1-4/ 2025-11-01 23:45:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4851 pages (at 41 pages/min), scraped 146 items (at 1 items/min) 2025-11-01 23:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/combination-wrenches/size~2-15-16/ 2025-11-01 23:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-punches/hole-diameter~1-21/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coolants/for-machining-type~milling/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-wire-marking-stickers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-grinder-accessories&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/thread-size~m16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ax103-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/for-hole-diameter~1-21&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coolants/for-machining-type~milling&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/thread-type~bspt/pipe-size~1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~t9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~m16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/automated-area-labels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nps-pipe-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=contact-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hole-punches/for-hole-diameter~1-21/ 2025-11-01 23:45:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5998243833923&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5998243833923%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145947%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000140314%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000033831%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000073588%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/drive-size~t9-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/thread-type~nps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=automated-area-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-mesh/material~bronze-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~10-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/drive-size~t9-1/ 2025-11-01 23:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-punches/hole-diameter~28-3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:46:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-mesh/material~brass-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4896 pages (at 45 pages/min), scraped 146 items (at 0 items/min) 2025-11-01 23:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002472444%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/thread-type~unf-1/pipe-size~3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002078531%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4656499217167&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20mesh&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4656499217167%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070378%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/milwaukee-batteries/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2602K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222602K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-mesh/material~bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/for-hole-diameter~28-3mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=control-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/thread-type~unf/pipe-size~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire-mesh/material~bronze/ 2025-11-01 23:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/size~11-16/ 2025-11-01 23:46:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hole-punches/for-hole-diameter~28-3mm/ 2025-11-01 23:46:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:46:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~29-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=milwaukee-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~18-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bakery-storage-racks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009285%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~ep4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-nut-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:46:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:46:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/hole-diameter~1-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~29-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/fitting-type~union-1/pipe-size~1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-belt-lacing/lacing-number~15s&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~18-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:46:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bakery-storage-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~ep4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/miniature-brass-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3430402569734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3430402569734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001669914%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/size~29-mm/ 2025-11-01 23:47:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6363923171247&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20belt%20lacing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6363923171247%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000126092%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/screwdrivers/size~18-mm/ 2025-11-01 23:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4944 pages (at 48 pages/min), scraped 147 items (at 1 items/min) 2025-11-01 23:47:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/screwdrivers/size~ep4/ 2025-11-01 23:47:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:47:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/hole-diameter~19-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/type~union/pipe-size~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/borosilicate-glass-rods-and-discs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6116K164&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226116K164%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miniature-brass-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3430402569734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3430402569734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002268827%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-belt-lacing/lacing-number~u4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086699%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/miniature-cam-followers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:47:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings-for-hydrochloric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-rope-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6363923171247&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20belt%20lacing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6363923171247%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000057851%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass/borosilicate-glass-rods-and-discs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/scribers/tip-style~flat/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6164K59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226164K59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9732413836877&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9732413836877%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%22220584230501%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miniature-cam-followers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings-for-nitrogen&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~t10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/banner-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8477K98&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228477K98%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-flanges/pipe-schedule~40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:47:58 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scribers/tip-style~flat&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~t90/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8358603820791&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8358603820791%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110513%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 4992 pages (at 48 pages/min), scraped 150 items (at 3 items/min) 2025-11-01 23:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~t10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8496026489774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8496026489774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002239703%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=banner-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-sticks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~0-157/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/socket-shape~12-point&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/size~t10/ 2025-11-01 23:48:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/socket-style~flex&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~t90&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barbers-shears/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8659884989431&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drive%20sockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8659884989431%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013173%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~1-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barrel-bung-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8659884989431&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drive%20sockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8659884989431%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002381313%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~0-1570&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/size~t90/ 2025-11-01 23:48:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:48:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:48:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coolant-hose-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~0-1570/ 2025-11-01 23:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barrel-liners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~16-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barbers%27-shears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~1-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-gaskets/pipe-size~20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barrel-bung-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-gaskets/pipe-size~4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:48:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~1-5-mm/ 2025-11-01 23:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~2-11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~2-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barrel-liners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~16-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5036 pages (at 44 pages/min), scraped 150 items (at 0 items/min) 2025-11-01 23:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/base-mount-magnifiers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-gaskets/for-pipe-size~20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-gaskets/for-pipe-size~4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~16-mm/ 2025-11-01 23:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~2-11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pipe-gaskets/for-pipe-size~20/ 2025-11-01 23:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~2-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pipe-gaskets/for-pipe-size~4/ 2025-11-01 23:49:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~2-11-16/ 2025-11-01 23:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-stripping-machines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~3-3-8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=base-mount-magnifiers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~2-5-8/ 2025-11-01 23:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/self-sealing-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/battery-chargers-and-starters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hones/sold-as~set&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=honeycomb-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/battery-contacts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1080139871402&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hones&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1080139871402%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002054337%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~3-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hook-and-loop-straps/material~polypropylene-fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=self-sealing-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-chargers-and-starters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~3-3-8/ 2025-11-01 23:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drum-dumpers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6020751333810&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hook%20and%20loop%20straps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6020751333810%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510787%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hook-racks/mount-type~magnetic-back/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cord-grips/knockout-trade-size~3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-contacts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drum-dust-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/battery-shunts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5084 pages (at 48 pages/min), scraped 150 items (at 0 items/min) 2025-11-01 23:50:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vibrating-feeders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/service-poles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:50:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hook-racks/mount-type~magnetic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mixers/for-use-with~oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mixers/for-use-with~citric-acid-10-concentration/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grips/for-knockout-trade-size~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/battery-terminal-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-shunts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mixers/electric-vibrating-feeders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wireless-receivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=service-poles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9998716923991&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hook%20racks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9998716923991%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003011040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cord-grips/for-knockout-trade-size~3-4/ 2025-11-01 23:50:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/mixers/electric-vibrating-feeders/ 2025-11-01 23:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mixers/for-use-with~oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mixers/for-use-with~citric-acid-10-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-terminal-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aspirin/manufacturer-brand~bayer/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sewer-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8018988495419&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8018988495419%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967526%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/mixers/for-use-with~citric-acid-10-/ 2025-11-01 23:50:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-sleeving&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drum-pump-liquid-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:50:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-augers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-biscuits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bbq-lighters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3529K46&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223529K46%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bayer-aspirin&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5129 pages (at 45 pages/min), scraped 150 items (at 0 items/min) 2025-11-01 23:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sewer-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hooks/mount-type~press-fit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bbq-lighters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6392083707491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hooks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6392083707491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drum-sand&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-cutting-circular-saw-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/voltage~24v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sheet-carts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=core-drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sheet-gasket-assortments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553708%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-locknuts/thread-type~nptf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-saws/diameter~5-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sheet-metal-clipping-machines/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-saws/diameter~55-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sheet-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4957971726222&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20locknuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4957971726222%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-screws/finish~black-oxide/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sheet-gasket-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mobil-polyrex-em-grease/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50785K146&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250785K146%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~5-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sheet-metal-clipping-machines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5176 pages (at 47 pages/min), scraped 151 items (at 1 items/min) 2025-11-01 23:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~55mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/for-use-with~apachi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drums/capacity~18-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-screws/finish~black-oxide-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hole-saws/diameter~55mm/ 2025-11-01 23:52:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shelving-pegs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179307%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobil-polyrex-em-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drums/capacity~1-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3968674775186&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drums&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3968674775186%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002135431%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wood-screws/finish~black-oxide-1/ 2025-11-01 23:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/beeswax/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drums/capacity~12-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=78365A11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2278365A11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3754K84&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223754K84%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bellows-shaft-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shelving-pegs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/belt-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drums/capacity-weight-~1-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=beeswax&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drums/capacity-weight-~12-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mobile-ladders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drums/capacity-weight-~1-gal-/ 2025-11-01 23:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3754K84&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223754K84%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bellows-shaft-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drums/capacity-weight-~12-gal-/ 2025-11-01 23:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/counters/fitting-connection~threaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drums/capacity~5-1-4-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=belt-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drums/capacity~50-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drums/capacity~550-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:53:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobile-ladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5222 pages (at 46 pages/min), scraped 153 items (at 2 items/min) 2025-11-01 23:53:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3754K84&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223754K84%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mobilgrease-xhp-221-grease/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:14 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=counters/electrical-connection-type~threaded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-054/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drums/capacity-weight-~5-1-4-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drums/capacity-weight-~50-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/countersinks/for-use-on~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drums/capacity-weight-~550-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/counters/electrical-connection-type~threaded/ 2025-11-01 23:53:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drums/capacity-weight-~5-1-4-gal-/ 2025-11-01 23:53:30 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:53:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drums/capacity-weight-~50-gal-/ 2025-11-01 23:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobilgrease-xhp-221-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drums/capacity-weight-~550-gal-/ 2025-11-01 23:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/for-use-on~bronze-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lags/for-use-in~blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shop-towels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-054inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-069/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/for-use-on~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bench-top-presses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=983458193638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=countersinks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B983458193638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929994%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-054inches/ 2025-11-01 23:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/countersinks/for-use-on~brass/ 2025-11-01 23:53:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-nipples-and-pipe/material~304-304l-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lags/for-use-in~block&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/modular-divider-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shop-towels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-069inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dry-box-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=525781165543&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20nipples%20and%20pipe&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B525781165543%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001547178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bench-top-presses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lags/for-use-in~block/ 2025-11-01 23:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5268 pages (at 46 pages/min), scraped 153 items (at 0 items/min) 2025-11-01 23:54:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-069inches/ 2025-11-01 23:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dry-erase-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dry-erase-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=modular-divider-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shoring-braces/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mold-release-agents/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bilge-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-nipples-and-pipe/pipe-size~3-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/fitting-connection~cam-and-groove&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=525781165543&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20nipples%20and%20pipe&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B525781165543%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000136564%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/for-use-on~ceramic-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shoring-braces&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002364232%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mold-release-agents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bilge-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~calcium-chloride&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=work-holding-turntables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bicycle-tires/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pastes/for-use-on~drywall/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-use-on~ceramic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:54:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/molybdenum-rectangular-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/molykote-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lubricants/for-use-on~ceramic/ 2025-11-01 23:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bicycle-tires&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5312 pages (at 44 pages/min), scraped 153 items (at 0 items/min) 2025-11-01 23:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=work-lights/mount-type~weighted-base&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-plugs/for-use-with~hydraulic-fluid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drywall-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sign-pole-banding/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=work-shoes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1685134723634&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=work%20lights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1685134723634%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010594%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8841143350592&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8841143350592%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964825%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=molybdenum-rectangular-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=molykote-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1920K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221920K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sign-pole-banding&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-242/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dual-line-air-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/momentary-rope-pull-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/signal-conditioning-relays/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/silica-gel-packs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-plugs/pipe-size~3-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/momentary-micro-switches already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:55:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/silicon-bronze-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8841143350592&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8841143350592%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000136564%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=momentary-rope-pull-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=signal-conditioning-relays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-242inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-251&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silica-gel-packs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5358 pages (at 46 pages/min), scraped 154 items (at 1 items/min) 2025-11-01 23:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-plugs/specifications-met~asme-b31-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-242inches/ 2025-11-01 23:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496084%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silicon-bronze-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/momentary-micro-switches already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=workbench-tops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/monel-wire/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-315/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8841143350592&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8841143350592%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002095111%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6100K358&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226100K358%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-plugs/specifications-met~astm-a276&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~krytox-240ab-type-ii/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-319&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8841143350592&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8841143350592%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145345%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-315inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:56:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=monel-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6100K358&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226100K358%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/monitors/electrical-connection~dvi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-335&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496691%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5288N49&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225288N49%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-315inches/ 2025-11-01 23:56:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~krytox-240ab-type-ii&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-plugs/specifications-met~din-908&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507282%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K736&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K736%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002088828%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~krytox-gpl-226/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8841143350592&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8841143350592%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002142246%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~lb-771-nickel-grade-anti-seize/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6100K358&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226100K358%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=monitors/connection-type~dvi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K735&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K735%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=10195K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2210195K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5405 pages (at 47 pages/min), scraped 155 items (at 1 items/min) 2025-11-01 23:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-34&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97313A107&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297313A107%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/monitors/connection-type~dvi/ 2025-11-01 23:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K736&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K736%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~krytox-gpl-226&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505092%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~lb-771-nickel-grade-anti-seize&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:21 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:57:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K735&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K735%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/monitors/electrical-connection~usb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/single-saddle-clips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-385/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099901%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K619&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K619%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002084155%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/blind-corner-sensors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K736&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K736%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=10195K151&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2210195K151%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=monitors/connection-type~usb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=single-saddle-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-385inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K735&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K735%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~lu-211-food-grade-dry-silicone/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blind-corner-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/monitors/connection-type~usb/ 2025-11-01 23:57:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:57:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-385inches/ 2025-11-01 23:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:57:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-railing-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/morris-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/blood-pressure-testers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct/connection-type~quick-disconnect-clamp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~lu-211-food-grade-dry-silicone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5454 pages (at 49 pages/min), scraped 159 items (at 4 items/min) 2025-11-01 23:58:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~lubriplate-no-130-a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-reducers/material~plastic-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/duct-size~20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~lubriplate-no-130-aa/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136299%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=morris-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blood-pressure-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=374262897454&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B374262897454%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001994913%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/duct-size~22&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~lubriplate-no-130-a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct/connection-type~quick-disconnect-clamp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:30 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/duct-size~20&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 23:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002014721%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~lubriplate-no-130-aa&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/blowgun-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~lubriplate-syn-lube-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002129252%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/motor-brackets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:40 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/duct-size~22&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-01 23:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002102167%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2289N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222289N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/finish~ptfe-coated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2289N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222289N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blowgun-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~lubriplate-syn-lube-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cubbies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=motor-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001554969%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:58:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002084597%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506107%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5500 pages (at 46 pages/min), scraped 161 items (at 2 items/min) 2025-11-01 23:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1016K86&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221016K86%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sliding-t-handle-drive-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=curtain-hardware&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-solder&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K757&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K757%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001554969%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-stays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-storage-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-strainers/material~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cubbies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~59-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~molykote-44-medium-high-temperature/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sliding-t-handle-drive-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001554969%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-strainers/material~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:59:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-936-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 23:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-support-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-swivels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pipe-strainers/material~stainless-steel/ 2025-11-01 23:59:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:59:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-921875inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolt-cutting-shears/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~molykote-44-medium-high-temperature&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498890%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-01 23:59:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 23:59:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-921875inches/ 2025-11-01 23:59:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 23:59:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121247%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolt-snap-assortments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8967K91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228967K91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct-ball-joints/gauge~18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-ball-joints/duct-size~10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5546 pages (at 46 pages/min), scraped 163 items (at 2 items/min) 2025-11-02 00:00:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/motors/motor-frame-size~nema-213t/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolt-cutting-shears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1252K27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221252K27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5434433323374&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct%20ball%20joints&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5434433323374%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002013718%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~1-217/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-thread-measuring-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~synxtreme-hd-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolt-snap-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-ball-joints/gauge~18-ga&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:25 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-ball-joints/duct-size~10&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=motors/motor-frame-size~nema-213t&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/duct-ball-joints/gauge~18-ga/ 2025-11-02 00:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~3-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~thk-afg/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~1-217inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9884332045743&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9884332045743%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001574186%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/motors/voltage~230v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~synxtreme-hd-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-threads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~1-217inches/ 2025-11-02 00:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089241%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/drive-size~3-1-8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-vibration-eliminators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~thk-afg&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/smoke-alarm-door-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mountable-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=motors/voltage~230v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1821A77&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221821A77%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:00:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/drive-size~3-1-8-1/ 2025-11-02 00:00:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089658%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9884332045743&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9884332045743%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553713%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5591 pages (at 45 pages/min), scraped 166 items (at 3 items/min) 2025-11-02 00:01:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/snacks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2317N2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222317N2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=smoke-alarm-door-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylinder-clevis-ends&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mountable-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~1-8-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cylinders/mount-type~pivot-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-fittings/duct-size~24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001494360%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/snap-toggles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=snacks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~1-39inches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8654208980817&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8654208980817%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002007480%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50315K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250315K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylindrical-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=d-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylinders/mounting-style~pivot&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~1-39/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~2-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:45 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-fittings/duct-size~24&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=snap-toggles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cylinders/mounting-style~pivot/ 2025-11-02 00:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506239%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50315K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250315K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=d-shackles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-124659-helical-inserts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/soil-pipe-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~1-39inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50415K56&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250415K56%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~2-7-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5637 pages (at 46 pages/min), scraped 167 items (at 1 items/min) 2025-11-02 00:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vacuum-breaking-valves/body-material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~2-76&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~1-39inches/ 2025-11-02 00:02:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506817%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50315K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250315K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-124659-helical-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-valve-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=soil-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50415K56&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250415K56%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50405K63&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250405K63%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-vacuum-breaking-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8989K661&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228989K661%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:02:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-124699-helical-inserts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-124709-helical-inserts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-valve-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50415K56&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250415K56%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nitrile-rubber-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/no-10-cardstock-tags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-15795-809-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-124699-helical-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=deep-hole-drills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-124709-helical-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/od~1-13-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nitrile-rubber-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5684 pages (at 47 pages/min), scraped 169 items (at 2 items/min) 2025-11-02 00:03:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=no-10-cardstock-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tig-torch-nozzles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-15795-809-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/finish~cadmium-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000162589%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-style~4-flute-spline/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brochure-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=513557860808&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B513557860808%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539197%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms16624-retaining-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/sae-hose-type~100r2at&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/tig-torch-nozzles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~mr-315-urethane-styrene-silicone-release-agent/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-viewing-spools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/for-drive-style~4-flute-spline&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334577%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brochure-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nozzles/tig-torch-nozzles/ 2025-11-02 00:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16624-1050-retaining-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/for-drive-style~4-flute-spline/ 2025-11-02 00:03:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-baskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/oil/trade-name~molykote-1000-thread-paste/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brush-dusters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~mr-315-urethane-styrene-silicone-release-agent&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bubble-mounts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334577%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bubble-packs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=detachable-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002112486%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pads/specifications-met~sae-ams3320/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-clamp-guards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5731 pages (at 47 pages/min), scraped 169 items (at 0 items/min) 2025-11-02 00:04:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~molykote-1000-thread-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brush-dusters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bubble-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334577%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bubble-packs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002125891%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002112486%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pads/specifications-met~sae-ams3320&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~0-141/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicator-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951811382783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951811382783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002091042%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bucket-tooth-plow-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:04:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/groove-coverage~full/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buffing-disc-polishing-felt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-test-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002112486%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~0-141inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951811382783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951811382783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002091042%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bucket-tooth-plow-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~0-141inches/ 2025-11-02 00:04:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/full-groove-dowel-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buffing-disc-polishing-felt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dust-cap-tethers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-16995-9-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fitting-lock-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-fittings/fitting-connection~barbed-2/for-use-with~dairy/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pins/full-groove-dowel-pins/ 2025-11-02 00:05:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:05:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5779 pages (at 48 pages/min), scraped 169 items (at 0 items/min) 2025-11-02 00:05:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~21-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951811382783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951811382783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002091042%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-fittings/fitting-connection~barbed-2/for-use-with~mild-bases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bull-points/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16995-9-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bullet-terminals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/connection-style~barbed/for-use-with~dairy&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dust-collector-hoods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~21millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dust-collector-vacuums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:05:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964932%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/connection-style~barbed/for-use-with~mild-bases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~21millimeters/ 2025-11-02 00:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/for-use-on~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spike-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bull-points&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bullet-terminals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-16997-102-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/material~plastic-1/material~polystyrene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179305%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=53415K261&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2253415K261%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/for-use-on~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spike-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16997-102-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/material~plastic-1/material~pvc-type-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/material~plastic/material~polystyrene-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~0-391-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5823 pages (at 44 pages/min), scraped 169 items (at 0 items/min) 2025-11-02 00:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~3millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/dies/for-use-on~iron/ 2025-11-02 00:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/fitting-connection~barbed-2/fitting-type~reducing-adapter&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plastic-sheets/material~plastic/material~polystyrene-plastic// 2025-11-02 00:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454885%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~3millimeters/ 2025-11-02 00:06:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dust-masks/niosh-approval-number~tc-84a-3174/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-16997-78-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~65-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002466710%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/material~plastic/material~pvc-type-2-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bus-bar-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5855N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225855N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plastic-sheets/material~plastic/material~pvc-type-2-plastic// 2025-11-02 00:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pneumatic-winches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/fitting-connection~barbed-2/for-use-with~hydrochloric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=digital-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spline-hammer-drills/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16997-78-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~65millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dust-masks/niosh-approval~tc-84a-3174&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002963068%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bus-bar-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~65millimeters/ 2025-11-02 00:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spline-key-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-winches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:06:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/dust-masks/niosh-approval~tc-84a-3174/ 2025-11-02 00:06:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5931N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225931N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyester-(petg)/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spline-hammer-drills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~ep14/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/butane-fuel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5866 pages (at 43 pages/min), scraped 171 items (at 2 items/min) 2025-11-02 00:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~ip2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spline-key-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/fitting-connection~barbed-2/shape~cross&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5931N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225931N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyester-(petg)&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polypropylene-pipe-nipples/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310771%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-funnels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=butane-fuel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~ep14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~ip2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/butcher-paper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/split-sleeve-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5218K862&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225218K862%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5931N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225931N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~ep14/ 2025-11-02 00:07:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~ip2/ 2025-11-02 00:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-pipe-nipples&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~no-17/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=butcher-paper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=split-sleeve-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/shape~cross&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/specifications-met~astm-b241&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dye-tablets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310771%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-hex-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002080228%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5916 pages (at 50 pages/min), scraped 172 items (at 1 items/min) 2025-11-02 00:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-nuts/military-specification~ms21044c04/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~no-17&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diodes/electrical-component-trade-number~1-5ke15a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-hollow-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5363K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225363K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~no-17/ 2025-11-02 00:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977684991444&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=diodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977684991444%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005391917%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~t25/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-21044-c04-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-inspection-mirrors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=material~a514-alloy-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=e-style-snap-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7632N45&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227632N45%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-reel-stops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diodes/electrical-component-trade-number~1n4730a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cabinet-liners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spreading-pliers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~t25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=100xl-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locknuts/military-specification~ms21045-7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977684991444&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=diodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977684991444%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005390995%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~t25/ 2025-11-02 00:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-stops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-loop-pipe-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-thresholds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7633N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227633N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cabinet-liners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spreading-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-casters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-cabinet-hinges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 5966 pages (at 50 pages/min), scraped 174 items (at 2 items/min) 2025-11-02 00:09:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-21045-7-locknuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-loaded-press-fit-ball-transfers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-casters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-mesh-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-cabinet-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diodes/electrical-component-trade-number~1n4755a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diodes/electrical-component-trade-number~1n5817&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diodes/electrical-component-trade-number~p6ke10ca&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-pellets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977684991444&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=diodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977684991444%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005391039%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977684991444&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=diodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977684991444%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005381603%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977684991444&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=diodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977684991444%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005392712%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=easy-drain-tanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-loaded-press-fit-ball-transfers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=easy-cut-strip-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7633N41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227633N41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7631N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227631N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7632N14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227632N14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrist-support-bands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-24671-3-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:09:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-pipe-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-24671-40-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-tempered-shim-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6015 pages (at 49 pages/min), scraped 177 items (at 3 items/min) 2025-11-02 00:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-24671-9-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7632N14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227632N14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eaton-snap-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wyes/fitting-connection~threaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24671-3-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24671-40-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wyes/connection-style~compression&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-tempered-shim-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24671-9-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wyes/fitting-connection~butt-weld&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wyes/connection-style~compression/ 2025-11-02 00:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7632N14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227632N14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wyes/connection-style~threaded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984228142725&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wyes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984228142725%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361746%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wyes/connection-style~threaded/ 2025-11-02 00:10:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screws/military-specification~ms24693-c294/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=disconnect-switches/switching-current~200-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=290xl-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=humidifier-valve-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=edge-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5237K452&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225237K452%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:10:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/id~1-5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5716559981042&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=disconnect%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5716559981042%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001584415%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=edge-controllers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/california-standard-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24693-c294-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6063 pages (at 48 pages/min), scraped 178 items (at 1 items/min) 2025-11-02 00:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flags/u-s-state-territory~wyoming/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/square-cone-sems-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/id~1-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5716559981042&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=disconnect%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5716559981042%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001584415%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=humidity-data-loggers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/square-cone-sems-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497107%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=divider-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=california-standard-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-24693-c74-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wyoming-flags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-cone-sems-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5716559981042&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=disconnect%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5716559981042%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001584415%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-cone-sems-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-24693-c96-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/camera-batteries/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24693-c74-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms27183-48/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-rolls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=edge-seals/for-use-with~acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/candles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24693-c96-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=camera-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:11:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=x-y-platforms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9900412680347&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=edge%20seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9900412680347%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984501%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-27183-48-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6112 pages (at 49 pages/min), scraped 178 items (at 0 items/min) 2025-11-02 00:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/performance~flame-retardant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=xlr-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/canton-flannel-buffs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=candles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8230254771998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20filaments&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8230254771998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002135098%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-chain-lanyards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/performance~static-control-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/color~fluorescent-blue&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-crimping-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-cylinder-manifolds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3665N411&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223665N411%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cap-nut-assortments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8230254771998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20filaments&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8230254771998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005316687%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=canton-flannel-buffs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002193612%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-chain-lanyards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-finishes-selector-packs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-dampers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-35207-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=85635K411&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2285635K411%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cap-nut-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-jack-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-finishes-selector-packs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-35335-58-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-drafts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35207-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6162 pages (at 50 pages/min), scraped 180 items (at 2 items/min) 2025-11-02 00:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-pallet-trucks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-pipe-benders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=egg-timers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-reservoirs/capacity~100-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stair-tread-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35335-58-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-35649-244-machine-screw-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hydraulic-reservoirs/flow-control-type~fixed/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eight-flute-end-mills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7305746640836&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hydraulic%20reservoirs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7305746640836%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101160%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/material~plastic-1/material~noryl-ppo&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carbon-steel-discs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:13:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-mounting-hardware-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001530052%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=62945K37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2262945K37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stair-tread-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35649-244-machine-screw-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-reservoirs/flow-control~fixed&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8561K419&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228561K419%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-steel-discs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/standard-nfpa-valve-mounting-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:13:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hydraulic-reservoirs/flow-control~fixed/ 2025-11-02 00:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/performance~flame-resistant-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:14:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/standoff-hangers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/performance~wear-resistant-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elastic-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6208 pages (at 46 pages/min), scraped 182 items (at 2 items/min) 2025-11-02 00:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/abrading-and-polishing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002094285%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standard-nfpa-valve-mounting-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-trucks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/performance-properties~flame-resistant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standoff-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=double-barbed-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrading-polishing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/plastic-sheets/performance-properties~flame-resistant/ 2025-11-02 00:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-trim&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/fitting-connection~grooved-end-clamp-on/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~0-007&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrading-polishing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/case-hardening-powders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shoulder-screws/military-specification~ms51575-9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001464277%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydrophobic-coatings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrading-polishing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abs-squares&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=double-sealed-ball-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/clamp-on-connection-type~grooved-end&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 500 Internal Server Error 2025-11-02 00:14:54 [scrapy.spidermiddlewares.httperror] (PID: 148) INFO: Ignoring response <500 https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrading-polishing&srchtxt=&mixedworld=force&workstationdata=true>: HTTP status code is not handled or not allowed 2025-11-02 00:14:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=case-hardening-powders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51575-9-shoulder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:14:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/elbows/clamp-on-connection-type~grooved-end/ 2025-11-02 00:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~0-035&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51576-4-shoulder-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-cylinder-rod-alignment-couplers/body-material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6256 pages (at 48 pages/min), scraped 182 items (at 0 items/min) 2025-11-02 00:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001453293%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/fitting-connection~quick-clamp-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51957-24-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drafting-stools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acme-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=87945K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2287945K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361293%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51576-4-shoulder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-air-cylinder-rod-alignment-couplers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/for-use-with~ethyl-alcohol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~16-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/caulk-guns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-24-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964443%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=zip%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001586981%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=87945K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2287945K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5713N66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225713N66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/for-use-with~iron-chloride&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51957-53-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caulk-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=draw-clasps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/for-use-with~krypton&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sleeve-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screws/military-specification~ms51957-68/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968005%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=87945K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2287945K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964505%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7436T211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227436T211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6308 pages (at 52 pages/min), scraped 183 items (at 1 items/min) 2025-11-02 00:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/for-use-with~sodium-carbonate-soda-ash/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-53-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-tote-baskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-68-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-spigots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drawer-slides/material~steel-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:16:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964505%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7436T211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227436T211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/for-use-with~sodium-carbonate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9864611406106&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drawer%20slides&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9864611406106%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511437%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-wall-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-tote-baskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-water-pressure-tanks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/elbows/for-use-with~sodium-carbonate/ 2025-11-02 00:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=impact-extensions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964505%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7436T211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227436T211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drawing-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-wall-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/material~bronze-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-water-pressure-tanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/material~bronze-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dremel-accessories&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-ties/length~4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:17:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6354 pages (at 46 pages/min), scraped 183 items (at 0 items/min) 2025-11-02 00:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000050743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/impact-wrenches/manufacturer~makita-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51958-80-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/material~bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51959-18-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~4-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/elbows/material~bronze/ 2025-11-02 00:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stack-lights/bulb-type~incandescent/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bit-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=impact-wrenches/manufacturer~makita&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51958-80-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/zip-ties/length~4-14/ 2025-11-02 00:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bit-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-ties/length~6-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51959-18-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/impact-wrenches/manufacturer~makita/ 2025-11-02 00:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51959-31-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/tube-od~7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tie-down-anchors/base-material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/tube-od~70-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=incandescent-stack-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cga-580-welding-gas-regulators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~6-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/pipe-size~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51959-31-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/for-tube-od~7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/zip-ties/length~6-ft-/ 2025-11-02 00:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/overall-length~5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6398 pages (at 44 pages/min), scraped 183 items (at 0 items/min) 2025-11-02 00:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-tie-down-anchors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/for-tube-od~70mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000067115%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cga-580-welding-gas-regulators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/adapters/for-tube-od~7-8/ 2025-11-02 00:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=zip%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000134488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/adapters/for-tube-od~70mm/ 2025-11-02 00:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~1-1-4-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/programmable-relays/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:25 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/overall-length~5-16&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51967-6-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/straightedge-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-link-fence-privacy-slats/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-link-fence-installation-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~3-8-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/thread-type~ght&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=programmable-relays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51967-6-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=industrial-chemicals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=straightedge-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000079141%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-link-fence-privacy-slats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastics/color~off-white-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-link-fence-installation-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3014N311&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223014N311%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chainmail-gloves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stretchable-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/multi-barbed-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6444 pages (at 46 pages/min), scraped 184 items (at 1 items/min) 2025-11-02 00:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/multicode-compatible-universal-transmitters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastics/color~off-white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastics/color~white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electric-dispensing-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chainmail-gloves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/checkpoint-recorders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=injection-quills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/plastics/color~off-white/ 2025-11-02 00:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=778212328374&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastics&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B778212328374%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stretchable-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=multi-barbed-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=multicode-compatible-universal-transmitters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/multipin-connector-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/prototyping-printer-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:37 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:19:37 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=checkpoint-recorders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastics/thickness~8-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/strut-channel-shelving-brackets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chip-boards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ptfe-angle-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inline-sanders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=778212328374&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastics&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B778212328374%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001463734%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=multipin-connector-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-insulation-jacketing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=prototyping-printer-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2656T66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222656T66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strut-channel-shelving-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chip-boards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:20:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6491 pages (at 47 pages/min), scraped 184 items (at 0 items/min) 2025-11-02 00:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ptfe-angle-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electric-motors/mounting-position~horizontal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chisel-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chloride-test-strips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ptfe-coated-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insertion-heaters-for-3d-printer-extruder-heating-elements/voltage~24v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chlorine-sensors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4889881527474&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electric%20motors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4889881527474%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126689%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/suction-feed-blasters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6194944965067&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insertion%20heaters%20for%203d%20printer%20extruder%20heating%20elements&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6194944965067%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553708%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chisel-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chloride-test-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ptfe-coated-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plates/material~aluminum-2/material~5052-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3715N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223715N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chlorine-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/manufacturer~loctite/manufacturer-model-number~088&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=suction-feed-blasters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9965629998276&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plates&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9965629998276%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511972%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nail-set-punches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002242151%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143624%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inserts/nut-type~screw-mount&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nails-for-nailers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ptfe-lubricants/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/narrow-jaw-vises/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9893535896715&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=inserts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9893535896715%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000165946%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chuck-keys/chuck-key-number~tk5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electric-winch-tuggers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6535 pages (at 44 pages/min), scraped 186 items (at 2 items/min) 2025-11-02 00:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nail-set-punches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nails-for-nailers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ptfe-lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pleated-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=narrow-jaw-vises&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nas-1149-cn816r-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chuck-keys/key-number~tk5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plcs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-boxes/cable-designation~seow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/circle-lasers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas1149-f0563p-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chuck-keys/key-number~tk5/ 2025-11-02 00:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/swage-standoffs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/swamp-cooler-pads/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:37 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/cigarette-butt-cans already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1149-cn816r-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-boxes/industry-designation~seow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=circle-lasers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1149-f0563p-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swage-standoffs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:52 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:21:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swamp-cooler-pads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/electrical-boxes/industry-designation~seow/ 2025-11-02 00:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plcs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inserts/thread-type~unjf-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:21:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/timer-relays/socket-connection~circular-pin-terminal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/cigarette-butt-cans already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9893535896715&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=inserts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9893535896715%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002393853%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-cabinets/environment~food-and-beverage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6578 pages (at 43 pages/min), scraped 186 items (at 0 items/min) 2025-11-02 00:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nas1351c-4-16-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pull-handles/mount-type~threaded-stud/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swedish-fish&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=circular-pin-timing-relays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-casters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-cylinders/mounting-position~horizontal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cabinets/environment-1~food-and-beverage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:25 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1351c-4-16-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switch-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pull-handles/mount-type~threaded-stud&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4216247920660&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4216247920660%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126689%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/electrical-cabinets/environment-1~food-and-beverage/ 2025-11-02 00:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1393964000724&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1393964000724%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010668%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-cylinders/rod-thread-size~1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/synthetic-leather/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-cylinders/rod-thread-size~3-4-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/syringe-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switch-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inspection-balls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plug-in-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:22:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4216247920660&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4216247920660%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000053773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4216247920660&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4216247920660%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000105365%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pulleys/application~for-lifting/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6491K216&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226491K216%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=synthetic-leather&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6239K347&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226239K347%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/cable-awg~16-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-groove-sockets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=syringe-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nas1352n-08-16-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6623 pages (at 45 pages/min), scraped 186 items (at 0 items/min) 2025-11-02 00:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562522%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pulleys/belt-trade-size~5vx/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/application~for-lifting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulated-socket-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nas1352n-4-8-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-groove-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation/flexibility~extra-flexible/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1352n-08-16-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8470423702633&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8470423702633%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002012536%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/cable-awg~18-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/for-belt-trade-size~5vx&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9466T75&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229466T75%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1352n-4-8-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562755%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/flexibility~very-flexible&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pulleys/for-belt-trade-size~5vx/ 2025-11-02 00:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/for-use-on~carriages&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K39&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K39%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/insulation/flexibility~very-flexible/ 2025-11-02 00:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/cable-awg~22-15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~15-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insulation&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930412%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pulleys/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001560914%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clips/clamp-type~u-bolt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496819%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6675 pages (at 52 pages/min), scraped 190 items (at 4 items/min) 2025-11-02 00:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/ethernet-cable-category~5e/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/fitting-type~flange&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nasm15795/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5648K49&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225648K49%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/material~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001502579%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002466432%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:24:21 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:24:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001560914%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clips/clamp-type~u-bolt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pulleys/material~brass/ 2025-11-02 00:24:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:24:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/for-cable-industry-designation~category-5e&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clothing-racks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9975836744491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9975836744491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001932189%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nasm-15795-819-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/plugs/for-cable-industry-designation~category-5e/ 2025-11-02 00:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pump-motor-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001560914%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/shape~half-round&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/cable-designation~nm-b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tachometers/tachometer-type~contact/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clothing-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~8-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9975836744491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9975836744491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001932189%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cnc-milling-machines/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insulation&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002308998%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003016158%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pump-motor-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497063%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:25:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1851N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221851N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6722 pages (at 47 pages/min), scraped 191 items (at 1 items/min) 2025-11-02 00:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=69935K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2269935K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tachometers/tachometer-type~contact&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tadpole-gaskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9975836744491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9975836744491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001932189%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cnc-milling-machines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pumps/power~1-12-hp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3018886232025&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3018886232025%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001979502%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tamper-evident-paint/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/for-use-with~lubricated-air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=11995T71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2211995T71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tadpole-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:30 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/for-use-with~salt-water&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965345%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/length~100-yd/length~300-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pumps/power~1-12-hp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tamper-evident-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=interchangeable-lock-cores&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964965%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=51495K171&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2251495K171%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001631801%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001616851%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3556277725899&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3556277725899%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001486368%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/material~bronze-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punch-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4373N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224373N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/material~ceramic-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000050743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/for-use-with~propane/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nema-10-20-receptacles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~1-3-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000160620%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6770 pages (at 48 pages/min), scraped 196 items (at 5 items/min) 2025-11-02 00:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/number-of-wires~8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punch-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/number-of-blades~2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~12-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001516753%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/for-use-with~propane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-10-20-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~1-3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~19-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/plug-number-of-blades~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~12mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/tap-extractors already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/punches/for-hole-diameter~1-3-16/ 2025-11-02 00:26:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001516753%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003266384%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/punches/for-hole-diameter~12mm/ 2025-11-02 00:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~20-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~19mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~22-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001516753%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum/material~aluminum-2/material~5456-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/tap-extractors already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:26:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/punches/for-hole-diameter~19mm/ 2025-11-02 00:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/material~aluminum-2/pressure-class~150/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~20mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-connection-lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:27:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/material~aluminum-2/pressure-class~3000-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6815 pages (at 45 pages/min), scraped 196 items (at 0 items/min) 2025-11-02 00:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~47-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~22-5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/punches/for-hole-diameter~20mm/ 2025-11-02 00:27:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511935%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/color-coded-feeler-gauges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-(class-150)-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/punches/for-hole-diameter~22-5mm/ 2025-11-02 00:27:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1306N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221306N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-(class-3000)-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/size~47-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=color-coded-feeler-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plumbers-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cord-circuit-interrupter-converters/certification~csa-certified&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/purell-hand-sanitizers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/collets/size~47-64/ 2025-11-02 00:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plunge-routers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5620190349820&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cord%20circuit%20interrupter%20converters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5620190349820%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002269638%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plungers/thread-size~1-4-28-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iso-32-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/push-button-air-directional-control-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-guide-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=purell-hand-sanitizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push%252fpull-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plunge-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~no-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plungers/thread-size~1-4-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6863 pages (at 48 pages/min), scraped 197 items (at 1 items/min) 2025-11-02 00:28:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-button-air-directional-control-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-cords/environment~washdown/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~no-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combined-counterbores-and-drills/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cords/cable-awg~16-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/plungers/thread-size~1-4-28/ 2025-11-02 00:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cords/environment~submersible&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7010901550041&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7010901550041%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561130%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/push-clips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7010901550041&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7010901550041%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000092646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/neon-indicator-light-bulbs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cords/environment~washdown-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cords/ul-file-number~e115330&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/temperature-markers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~no-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combined-counterbores-and-drills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7248K25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227248K25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/electrical-cords/environment~washdown-1/ 2025-11-02 00:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7010901550041&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7010901550041%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005167358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/combination-wrenches/size~no-8/ 2025-11-02 00:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/push-in-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=neon-indicator-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70355K701&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270355K701%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/push-lock-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temperature-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ballasts/light-technology~cfl/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jackets/performance~metal-detectable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:28:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/neoprene-round-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6906 pages (at 43 pages/min), scraped 199 items (at 2 items/min) 2025-11-02 00:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-in-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stickers/message~push/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-lock-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compact-fluorescent-ballasts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jackets/performance-properties~metal-detectable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cords/electrical-connection~pin-and-sleeve&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=neoprene-round-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/jackets/performance-properties~metal-detectable/ 2025-11-02 00:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-rods/material~aluminum-2/material~6101-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-sheets/material~aluminum-2/material~2024-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-hacksaw-saws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7010901550041&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7010901550041%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003069963%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-sheets/material~aluminum-2/material~5005-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-stickers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5481N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225481N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-breaker-poles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/material~aluminum/material~2024-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-rods/material~aluminum-2/material~6101-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nestable-trays/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/material~aluminum/material~5005-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510120%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-lifts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/compression-spring-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-lines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510051%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/push-to-connect-tube-fittings-for-rigid-white-polypropylene-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=88835K611&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2288835K611%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6951 pages (at 45 pages/min), scraped 200 items (at 1 items/min) 2025-11-02 00:30:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nestable-trays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6062K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226062K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-rods/material~aluminum-2/material~6101-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flags/u-s-state-territory~new-hampshire/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compression-spring-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-plugs/certification~ul-recognized-component&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512093%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-to-connect-tube-fittings-for-rigid-white-polypropylene-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=88835K611&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2288835K611%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/thickness~0-04&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6371003242508&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6371003242508%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002256898%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nickel-alloy-400-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=new-hampshire-flags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/putty/for-use-on~bronze-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001457391%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/thickness~0-094&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nickel-alloys/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=88835K611&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2288835K611%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thermoplastic-ductwork/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nickel-alloy-400-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nickel-bar-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001458394%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nickel-copper-alloy-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=putty/for-use-on~bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:30:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nickel-alloys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermoplastic-ductwork&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/putty/for-use-on~bronze/ 2025-11-02 00:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thermostat-test-kits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 6997 pages (at 46 pages/min), scraped 201 items (at 1 items/min) 2025-11-02 00:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nickel-bar-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nickel-copper-alloy-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:31:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pocket-rulers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/condiment-dispensers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conductive-uhmw-polyethylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pocket-scissors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermostat-test-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-wire/csa-file-number~013427-0-000&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pocket-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conductivity-probes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4072418438324&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4072418438324%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005190343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-thumb-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=condiment-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conductive-uhmw-polyethylene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-clevises/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8209K31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228209K31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvc-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conductivity-probes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pole-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/fitting-connection~quick-disconnect/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvc-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-clevises&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cone-grinding-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7044 pages (at 47 pages/min), scraped 202 items (at 1 items/min) 2025-11-02 00:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-insulating-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/confor/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/for-use-with~boric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/connection-style~quick-disconnect&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=poles/material~mica&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connector-pliers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cone-grinding-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nipples/connection-style~quick-disconnect/ 2025-11-02 00:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-and-lighting/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992979646156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=poles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992979646156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512602%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=confor&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~boric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thrust-needle-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conspicuity-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5067K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225067K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connector-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=emergency-equipment-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964962%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvc-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-lighting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jars/capacity~1-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/contact-cement/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conspicuity-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=53505K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2253505K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-lighting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thrust-needle-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jars/capacity~175-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-lighting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7087 pages (at 43 pages/min), scraped 203 items (at 1 items/min) 2025-11-02 00:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jars/capacity-weight-~1-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polished-copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jars/capacity~2-3-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=contact-cement&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:16 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 500 Internal Server Error 2025-11-02 00:33:16 [scrapy.spidermiddlewares.httperror] (PID: 148) INFO: Ignoring response <500 https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-lighting&srchtxt=&mixedworld=force&workstationdata=true>: HTTP status code is not handled or not allowed 2025-11-02 00:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=53505K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2253505K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/jars/capacity-weight-~1-gal-/ 2025-11-02 00:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvc-pipe-reducing-tees/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jars/capacity-weight-~175-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/for-use-with~propane/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/for-use-with~sulfuric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jars/capacity~360-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/jars/capacity-weight-~175-ml/ 2025-11-02 00:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jars/capacity-weight-~2-3-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/for-use-with~vegetable-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=53505K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2253505K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-pipe-reducing-tees&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-plugs/for-wire-gauge~17&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/jars/capacity-weight-~2-3-ml/ 2025-11-02 00:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jars/capacity~5000-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~propane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~sulfuric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jars/capacity-weight-~360-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6371003242508&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6371003242508%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001992786%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~vegetable-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:33:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/jars/capacity-weight-~360-ml/ 2025-11-02 00:33:57 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvc-u-channels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jars/capacity-weight-~5000-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polishing-wheel-cement&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8798T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228798T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965616%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7133 pages (at 46 pages/min), scraped 203 items (at 0 items/min) 2025-11-02 00:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/jars/capacity-weight-~5000-ml/ 2025-11-02 00:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tire-jacks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coolant-guns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ansi-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-u-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/copper-hammers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tire-jacks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/toggle-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polycarbonate-round-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coolant-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jib-arms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvdf-sanitary-quick-clamp-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=arc-flash-protection-jackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=toggle-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/thread-type~bspp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvdf-sanitary-quick-clamp-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyester-(pet)-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:34:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hammer-drills/power-source~battery/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-resistance-strain-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aro-shape-hose-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7176 pages (at 43 pages/min), scraped 204 items (at 1 items/min) 2025-11-02 00:35:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quad-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/thread-type~bspp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/corkscrews/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-sockets/connector-shell-size~10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cordless-hammer-drills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/for-use-on~polyurethane-rubber-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000037076%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tool-and-die-shelving/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyester-(pet)-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8358313672822&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20sockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8358313672822%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001566834%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quad-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929514%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corkscrews&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6134T63&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226134T63%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:35:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxy-needles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/corner-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tool-and-die-shelving&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:42 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:35:47 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jt3-chuck-keys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:47 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quick-clamp-butterfly-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quick-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alligator-belt-lacing/lacing-number~20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corner-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-angles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=axles/diameter~30-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-duct-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-clamp-butterfly-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kaizen-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7224 pages (at 48 pages/min), scraped 205 items (at 1 items/min) 2025-11-02 00:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681008859035&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=axles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681008859035%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000160942%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kapton-dots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=no-20-alligator-belt-lacing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1482K49&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221482K49%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alligator-belt-lacing/lacing-number~35n/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bags/capacity~15-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=evacuation-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quick-disconnect-fan-power-cords/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quick-disconnect-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/no-engine-idling-signs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kerosene-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=no-35n-alligator-belt-lacing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bags/capacity-weight-~15-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kevlar%252fnylon-rounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-wire/cable-designation~xhhw-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-disconnect-fan-power-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-disconnect-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=no-engine-idling-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bags/capacity-weight-~15-gal-/ 2025-11-02 00:36:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-rounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:36:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4072418438324&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4072418438324%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003015077%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crack-chasers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quick-disconnect-tube-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7270 pages (at 46 pages/min), scraped 206 items (at 1 items/min) 2025-11-02 00:37:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7715T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227715T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/no-u4-clipper-belt-lacing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-screw-nuts/thread-direction~right-hand&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kevlar-nylon-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=expanding-collars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7489222468978&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ball%20screw%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7489222468978%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000138654%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-totes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crack-chasers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-disconnect-tube-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7715T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227715T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=no-u4-clipper-belt-lacing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/quick-exhaust-relief-valves already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=expansion-joints/food-industry-standard~nsf-ansi-61&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7715T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227715T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nonmagnetic-retaining-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balls/diameter~3-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balls/diameter~5-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=key-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-turn-polypropylene-tube-fitting-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7692142082491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=expansion%20joints&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7692142082491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002097343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/quick-exhaust-relief-valves already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/flammability-rating~ul-2556-vw-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=balls&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000077724%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=balls&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001531232%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5304N18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225304N18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:00 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonmagnetic-retaining-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002380810%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hasps/hasp-type~keyed/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7319 pages (at 49 pages/min), scraped 206 items (at 0 items/min) 2025-11-02 00:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/trailer-barriers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9292K74&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229292K74%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extension-cords/length~6-ft/length~72/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nonmarring-sledges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:17 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/transfer-needles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keyed-hasps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trailer-barriers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/length~6-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rack-post-protectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonmarring-sledges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/extension-cords/length~6-ft-/ 2025-11-02 00:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electricians-troubleshooting-and-testing-pocket-guide-books/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-felt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=transfer-needles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/serial-port-interface~rs-232&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rack-post-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rack-protectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~36-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001567256%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-film&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electricians-troubleshooting-%26-testing-pocket-guide-books&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crown-gears/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rack-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7364 pages (at 45 pages/min), scraped 208 items (at 2 items/min) 2025-11-02 00:39:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/size~36-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/battery-size~31&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~1092r/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001567256%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/battery-size~n&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/crowfeet/size~36-mm/ 2025-11-02 00:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003030357%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crown-gears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003030922%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-line&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/racks/component~frame-hanger/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/travel-limiting-lanyards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrodes/aws-material-code~er308lsi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~1092r&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001567256%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994383438024&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994383438024%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~1092r/ 2025-11-02 00:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~a1001eh/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~a1011d41/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=racks/component~frame-hanger&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrodes/for-joining~aluminum-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=travel-limiting-lanyards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:47 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrodes/aws-material-code~er308lsi&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nutdrivers/performance~nonsparking-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:39:52 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=684483403398&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B684483403398%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243426%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994383438024&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994383438024%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453189%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/culture-tubes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~a1001eh&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/current-data-loggers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~a1011d41&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7414 pages (at 50 pages/min), scraped 208 items (at 0 items/min) 2025-11-02 00:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonsparking-nutdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~a1001eh/ 2025-11-02 00:40:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~a1011d41/ 2025-11-02 00:40:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/current-transformers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tripod-stands/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=684483403398&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B684483403398%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243426%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=culture-tubes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eye-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rags/rag-type~huck-towel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=current-data-loggers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cut-resistant-footwear/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=current-transformers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tripod-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rail-mounts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=684483403398&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B684483403398%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243426%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rail-ends/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rags/rag-type~huck-towel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fabric-punches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cut-resistant-footwear&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5853343975084&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5853343975084%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001973483%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electronic-protractors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/noryl-round-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/truss-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rail-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rail-ends&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/railroad-signs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rails/component~cap/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/snap-loc-coolant-hose/snap-loc-trade-size~3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7458 pages (at 44 pages/min), scraped 208 items (at 0 items/min) 2025-11-02 00:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/abrasive-blasting-nozzles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=noryl-round-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cylinder-sleeves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=truss-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fan-shrouds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-knife-positioning-arms/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=railroad-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rails/component~cap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/3-4-snap-loc-coolant-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/abrasive-blasting-nozzles-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylinder-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cylinder-supports/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/d-coded-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9948522430991&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9948522430991%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002253666%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%227729056834350%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/adjustable-mounting-arms-for-air-knives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nozzles/abrasive-blasting-nozzles-4/ 2025-11-02 00:41:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~3203&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1207A18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221207A18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/mounting-location~wall&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5307K96&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225307K96%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nozzles/adjustable-mounting-arms-for-air-knives/ 2025-11-02 00:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rails/component~track-mounting-bolt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~sc4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002111211%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylinder-supports&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=d-coded-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002266240%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:41:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rails/component~trolley/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7505 pages (at 47 pages/min), scraped 210 items (at 2 items/min) 2025-11-02 00:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rails/component~track-mounting-bolt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~sc4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~604&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-wrap/material~polyurethane-rubber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ramps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rails/component~trolley&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/fastener-head-type~plain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9948522430991&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9948522430991%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002276307%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~sc4/ 2025-11-02 00:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=embossing-stamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143956%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6303&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9948522430991&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9948522430991%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002277425%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001595930%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9484T71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229484T71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyurethane-wire-wrap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002150707%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ramps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:37 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:42:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/manufacturer-equivalent~national/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6304&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002084831%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/thread-size~0-586-32-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/turn-button-fastener-bases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002150707%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pooper-scoopers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~0-586-32-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-manufacturer~national&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pool-shock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pop-rivets/mandrel-material~aluminum-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449209%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321592%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~0-586-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7557 pages (at 52 pages/min), scraped 211 items (at 1 items/min) 2025-11-02 00:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pop%20rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002217335%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=turn-button-fastener-bases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002150707%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6804&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/fasteners/thread-size~0-586-32/ 2025-11-02 00:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=emery-cloth-rolls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nozzles/spray-pattern~multiple-stream/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002118751%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/deodorizers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pop-up-canopies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:43:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tank-washing-nozzles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~684&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tweezer-cases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=porous-foam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002151585%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/spray-pattern~multiple-stream&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/desk-dividers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=deodorizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~696&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/stationary-tank-washing-nozzles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000052108%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fdm-printer-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002074153%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:43:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nozzles/stationary-tank-washing-nozzles/ 2025-11-02 00:44:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezer-cases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ratchets/power-source~battery/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=10095K84&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2210095K84%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/twisted-o-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=desk-dividers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7604 pages (at 47 pages/min), scraped 211 items (at 0 items/min) 2025-11-02 00:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/desk-trays/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tig-torches/cooling-method~water/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/desktop-axial-fans/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ratchets/power-source~battery&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=twisted-o-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/electrical-connection~rca/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=desk-trays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9988466368360&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9988466368360%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076444%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reagent-nitric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=enclosed-relays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/water-cooled-tig-torches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~1092nr/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=desktop-axial-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nozzles/water-cooled-tig-torches/ 2025-11-02 00:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/type-6-web-slings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rca-cable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/diamond-core-drills/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reagent-nitric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~1176/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~1092nr&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=enclosure-drains&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=portable-stairs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/flute-style~expandable/reamer-operation~hand/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=type-6-web-slings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=portable-tents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diamond-core-drills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~1092nr/ 2025-11-02 00:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7649 pages (at 45 pages/min), scraped 212 items (at 1 items/min) 2025-11-02 00:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~1176&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:45:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/u-type-fasteners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~125-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nut-gauges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~1176/ 2025-11-02 00:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=type-thhn%252fthwn-building-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/expansion-hand-reamers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/reamers/expansion-hand-reamers/ 2025-11-02 00:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=enclosures/material~carbon-fiber-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~19-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=u-type-fasteners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~125mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nut-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/digital-counting-scales/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~19-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~210-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4443870537041&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=enclosures&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4443870537041%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003423553%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000051516%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/for-housing-id~125mm/ 2025-11-02 00:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000169573%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000075290%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1935N6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221935N6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/nut-type~locknut/fastener-strength-grade-class~iso-class-6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=post-label-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=digital-counting-scales&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/underwear/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~4-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~280&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000075290%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7701 pages (at 52 pages/min), scraped 213 items (at 1 items/min) 2025-11-02 00:46:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/fastener-strength-grade-class~class-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000141370%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401179%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=field-wireable-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=posts/material~plastic-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=underwear&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~5-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1189A32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221189A32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=posts/mounting-location~in-ground&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000075290%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9967987003482&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=posts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9967987003482%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000113467%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9967987003482&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=posts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9967987003482%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002380228%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/universal-socket-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-0640/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/nut-type~locknut/thread-size~1-2-20-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1189A32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221189A32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:46 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:46:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~65-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~3408&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000113467%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=universal-socket-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0640&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/direct-to-metal-coatings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~1-2-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:46:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401100%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1189A32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221189A32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/cut-style~gear&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000162909%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~65mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7751 pages (at 50 pages/min), scraped 213 items (at 0 items/min) 2025-11-02 00:47:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=11775A806&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2211775A806%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000113467%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~70-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002240023%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=direct-to-metal-coatings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/for-housing-id~65mm/ 2025-11-02 00:47:16 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:47:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0640&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:47:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:47:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3034A313&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223034A313%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/usb-expansion-hubs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/shaft-diameter~1-3-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~3769&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:30 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:47:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~70mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400599%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/disposable-coats/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pouch-laminators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/for-housing-id~70mm/ 2025-11-02 00:47:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~53684&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3034A313&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223034A313%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=usb-expansion-hubs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=poulan-compatible-chain-saw-chains&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-shaft-diameter~1-3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=11775A807&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2211775A807%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/shaft-diameter~10-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400917%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:47:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:47:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/for-shaft-diameter~1-3-16/ 2025-11-02 00:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=disposable-coats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12095A191&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212095A191%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~997x/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3034A313&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223034A313%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7796 pages (at 45 pages/min), scraped 215 items (at 2 items/min) 2025-11-02 00:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-groove-milling-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~a1114fm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-shaft-diameter~10mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/material~steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=powdered-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/for-shaft-diameter~10mm/ 2025-11-02 00:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12095A191&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212095A191%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~997x&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:24 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-groove-milling-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~0-021&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~a1114fm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/distance-measuring-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~997x/ 2025-11-02 00:48:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001701921%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~a1114fm/ 2025-11-02 00:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12095A191&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212095A191%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/nut-type~locknut/thread-type~unc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~a389&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dock-door-seals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8832A37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228832A37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:51 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dock-plates/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~a506&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~0-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400506%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vacuum-evaporation-boats/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-type~unc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401458%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001703296%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7843 pages (at 47 pages/min), scraped 216 items (at 1 items/min) 2025-11-02 00:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dock-door-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/doctors-scales/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002102665%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-cords/iec-connector-type~c5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dock-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5000N17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225000N17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-evaporation-boats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~1-1-4-7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vacuum-sealers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=doctors%27-scales&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-cords/international-power-connector-standard~as-nzs-3112&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-cords/iec-style~c5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-cords/international-power-connector-standard~cee-7-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~1-173-18-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4895046738192&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=power%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4895046738192%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448797%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/power-cords/iec-style~c5/ 2025-11-02 00:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4895046738192&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=power%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4895046738192%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455600%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-1-4-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~11-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-sealers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-cords/voltage~240v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/end-mills/mill-diameter~12-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1447K13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221447K13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/utility-knives/blade-type~fixed/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504347%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4895046738192&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=power%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4895046738192%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553777%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:49:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-173-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4895046738192&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=power%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4895046738192%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448797%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:50:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nuts/thread-size~1-173-18/ 2025-11-02 00:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7892 pages (at 49 pages/min), scraped 218 items (at 2 items/min) 2025-11-02 00:50:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~co89/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~12-5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fixed-blade-utility-knives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~in3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fixed-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-1600/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4895046738192&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=power%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4895046738192%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448797%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/end-mills/mill-diameter~12-5mm/ 2025-11-02 00:50:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:50:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/material~ptfe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~2-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~co89&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510788%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~in3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001718473%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1600&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:38 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:50:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~co89/ 2025-11-02 00:50:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~21-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~3-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~in3/ 2025-11-02 00:50:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3034A135&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223034A135%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vci-(volatile-corrosion-inhibitor)-bags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163521%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~2-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001718785%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/door-strike-plates already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/end-mills/mill-diameter~2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000109643%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:50:56 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1600&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3051A364&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223051A364%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flange-locknuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~98-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-1750/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vci-(volatile-corrosion-inhibitor)-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~2-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7937 pages (at 45 pages/min), scraped 220 items (at 2 items/min) 2025-11-02 00:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~2mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/double-end-pin-gauges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/door-strike-plates already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451477%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/end-mills/mill-diameter~2mm/ 2025-11-02 00:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~98mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1750&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:26 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~2-12&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:51:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/shaft-diameter~1-7-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~5-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=double-end-pin-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/od~98mm/ 2025-11-02 00:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000071487%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/shaft-diameter~1-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/double-v-belt-pulleys/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002249857%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001706279%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:38 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanged-spacers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:51:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-1945/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:40 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1750&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:51:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002263047%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8876A31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228876A31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/end-mills/mill-diameter~9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=double-v-belt-pulleys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-2015/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-2150/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1945&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~9-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~p139&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/shaft-diameter~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 7985 pages (at 48 pages/min), scraped 221 items (at 1 items/min) 2025-11-02 00:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~9mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000093782%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~ru53/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001710334%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2015&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2150&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/shaft-diameter~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000140902%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/end-mills/mill-diameter~9mm/ 2025-11-02 00:52:17 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1945&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:52:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8207A473&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228207A473%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000059161%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000084401%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002241598%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~ru53&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:31 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2015&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:33 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2150&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=engine-cranes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flappers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~ru53/ 2025-11-02 00:52:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8207A473&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228207A473%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/downspouts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-2205/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=engine-driven-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drafting-compasses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~m26/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8207A473&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228207A473%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=downspouts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2205&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drafting-compasses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~sm737&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8035 pages (at 50 pages/min), scraped 221 items (at 0 items/min) 2025-11-02 00:53:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m26&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~sm936&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000016358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:14 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flat-hand-files&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401475%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333485%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:21 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vise-handles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:24 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2205&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-2497/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-2520/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epdm-elastic-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/visually-impaired-mats/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:53:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/manufacturer~abloy-security&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2497&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bit-socket-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pps-push-to-connect-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2520&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002233380%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bit-racks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000080737%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wafer-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=visually-impaired-mats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000073012%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epoxies/color~green-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8082 pages (at 47 pages/min), scraped 221 items (at 0 items/min) 2025-11-02 00:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-2795/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:09 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2520&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bit-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wafer-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-fabric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blowers/outlet-diameter~6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/width~10-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/color~green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2795&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-flow-adjustment-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/fastener-head-type~oval/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9955125964573&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=blowers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9955125964573%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000075689%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000123280%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/epoxies/color~green/ 2025-11-02 00:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-hex-drivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000035204%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~1-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-insert-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:45 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2795&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:54:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:54:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:54:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/head-type~oval&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9955125964573&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=blowers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9955125964573%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000075689%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:54:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keystone-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A754&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A754%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=float-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bolts/head-type~oval/ 2025-11-02 00:55:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-insert-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-spacers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kickstands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8133 pages (at 51 pages/min), scraped 221 items (at 0 items/min) 2025-11-02 00:55:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:55:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9955125964573&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=blowers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9955125964573%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000075689%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/waste-container-dollies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-straightedges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~1-7-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-test-weights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-fillers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/for-use-on~stone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:35 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kitchen-knives/knife-type~produce&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=waste-container-dollies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930162%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kitchen-sinks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6457366239264&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=kitchen%20knives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6457366239264%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968693%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~1-7-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~5-16-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~11-3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7053A12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227053A12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~1-7-mm/ 2025-11-02 00:55:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~11-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~10760&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334815%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930162%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143061%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8183 pages (at 50 pages/min), scraped 223 items (at 2 items/min) 2025-11-02 00:56:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~11-3-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~11c&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~11-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74575A7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274575A7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-control-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930162%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~11-3-mm/ 2025-11-02 00:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089966%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-control-orifices&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~11-32/ 2025-11-02 00:56:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 00:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-fit-framing-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1813A222&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221813A222%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-sheet-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74575A7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274575A7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:30 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/press-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-tie-down-strap-components/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-tipped-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-type~uns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~2-6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/press-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/drill-bits/size~13-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74575A7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274575A7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107915%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knives/for-use-on~wood-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-tipped-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-tie-down-strap-components&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~2-6-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-twine/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2846472700070&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=knives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2846472700070%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930270%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:56:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:57:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~2-6-mm/ 2025-11-02 00:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/drill-bits/size~13-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8233 pages (at 50 pages/min), scraped 224 items (at 1 items/min) 2025-11-02 00:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~2-85-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-3645/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~22/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-twine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-controls/for-use-with~soap-solutions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=boxes/shape~square&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/finish~anodized&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:57:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~2-85-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9072876622555&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20controls&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9072876622555%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965378%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knobs/mount-type~adhesive-back&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3645&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~22-ga&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995901722036&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=boxes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995901722036%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310398%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=brackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550239%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~2-85-mm/ 2025-11-02 00:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8323758595347&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=knobs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8323758595347%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010855%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000060803%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~22-ga/ 2025-11-02 00:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-fit-nuts/pem-part-number~cls-0616-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~27/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~e-30cl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9072876622555&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20controls&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9072876622555%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965378%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2434N2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222434N2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:50 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3645&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 00:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9946404817603&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=press%20fit%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9946404817603%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114799%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~3-2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brackets/screw-size~no-6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002078173%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-fit-nuts/pem-part-number~cls-256-0&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~27-ga&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6430A23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226430A23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9072876622555&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20controls&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9072876622555%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965378%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8281 pages (at 48 pages/min), scraped 225 items (at 1 items/min) 2025-11-02 00:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/mount-type~freestanding&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9946404817603&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=press%20fit%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9946404817603%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110138%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~3-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/for-screw-size~no-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~27-ga/ 2025-11-02 00:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxy-adhesive-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=brackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010917%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxy-anchor-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~3-2-mm/ 2025-11-02 00:58:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/brackets/for-screw-size~no-6/ 2025-11-02 00:58:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~010/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wedge-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxy-coating-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-valves/valve-type~ball/pipe-size~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/reamers/reamer-diameter~0-3930/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~029/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knuckle-threads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/screw-size~no-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=339191585183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B339191585183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247047%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000115106%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kool-mist-formula-78-coolants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~010&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~033/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wedge-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:58:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=brackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000140022%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~036/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~029&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/reamers/reamer-diameter~0-3930/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8326 pages (at 45 pages/min), scraped 226 items (at 1 items/min) 2025-11-02 00:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~116/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000079420%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~033&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=brackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~036&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-fit-nuts/pem-part-number~s-m3-1zi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000076246%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=er-20-collets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-barbed-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:27 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000065414%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 00:59:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9946404817603&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=press%20fit%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9946404817603%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136582%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000079420%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K121&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K121%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=brackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K124&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K124%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~116&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-fit-nuts/pem-part-number~ss-024-0zi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-4325/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~74/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000133726%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9946404817603&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=press%20fit%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9946404817603%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002082459%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~136/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000079420%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 00:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=er-70s-6-tig-welding-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 00:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4325&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~8-9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~74-ga&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-compression-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8375 pages (at 49 pages/min), scraped 228 items (at 2 items/min) 2025-11-02 01:00:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-4405/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~136&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000083688%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~74-ga/ 2025-11-02 01:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~142/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000057990%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:21 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4325&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~177/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~8-9-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4405&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-4595/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-industry-plastics&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~142&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~8-9-mm/ 2025-11-02 01:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=folding-arms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000135062%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=esd-wipes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~177&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000053941%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4595&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:50 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4405&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:00:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:00:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laboratory-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=etching-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000159250%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K148&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K148%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000131188%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-point-gauges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=etfe-film&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K335&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K335%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8424 pages (at 49 pages/min), scraped 229 items (at 1 items/min) 2025-11-02 01:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=footwear&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:08 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4595&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-and-vacuum-transmitters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/welding-wire-cases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drilled-shafts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wet-location-flush-mount-machine-lights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~264/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-4910/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ethernet-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/strut-channel/strut-channel-type~high-profile/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-wire-cases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drilled-shafts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wet-location-flush-mount-machine-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=frames&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~264&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/for-use-with~acetylene-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4910&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wheel-chocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000124330%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing/high-profile-strut-channel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7534361604797&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pressure%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7534361604797%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968999%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038444%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~312/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K203&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K203%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971113029698&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=framing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971113029698%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%225684901981911%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=32255K7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2232255K7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheel-chocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lag-anchors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ladder-to-scaffold-conversion-supports&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8470 pages (at 46 pages/min), scraped 230 items (at 1 items/min) 2025-11-02 01:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3548T211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223548T211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038444%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~332/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~312&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=32255K7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2232255K7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/for-use-with~wastewater-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~345/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000074406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lags/for-use-in~hollow-bricks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038444%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~332&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-fan-guards/blade-diameter~24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K385&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K385%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/window-blinds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-fan-guards/blade-diameter~42&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9974560404121&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lags&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9974560404121%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929824%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~345&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000113955%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5195649950898&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=exhaust%20fan%20guards&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5195649950898%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000072575%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=32255K7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2232255K7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/for-use-with~wastewater&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5195649950898&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=exhaust%20fan%20guards&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5195649950898%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000040106%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:52 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000015203%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K52&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K52%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19555K14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219555K14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:02:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pressure-gauges/for-use-with~wastewater/ 2025-11-02 01:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-blinds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~e6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19555K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219555K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tabletops/material~laminate/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:03:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8520 pages (at 50 pages/min), scraped 235 items (at 5 items/min) 2025-11-02 01:03:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~e7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/window-tint/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~386/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~e6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/window-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laminate-tabletops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-fans/blade-diameter~11-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-fans/blade-diameter~18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~e7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~e6/ 2025-11-02 01:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19555K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219555K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuel-transfers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9102215783954&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=exhaust%20fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9102215783954%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005107602%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9102215783954&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=exhaust%20fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9102215783954%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000144151%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-tint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~e7/ 2025-11-02 01:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~386&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~406/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2232K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222232K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19455K33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219455K33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000073575%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-6900/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19555K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219555K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~t30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K488&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K488%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lamp-installation-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~406&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lamp-safety-shields&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8566 pages (at 46 pages/min), scraped 237 items (at 2 items/min) 2025-11-02 01:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-6900&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000092471%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:14 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:04:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/maximum-pressure~8-psi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=furniture-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~t30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-pipe-elbows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000167995%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/diameter~2-9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-8400/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K502&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K502%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~t30/ 2025-11-02 01:04:29 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-6900&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drop-ceiling-lasers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/maximum-pressure-psi~8-psi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-8850/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~2-9mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/measurement-unit~decanewtons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-8400&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7534361604797&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pressure%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7534361604797%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001703189%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-thrust-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-tooth-lock-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/diameter~2-9mm/ 2025-11-02 01:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000088392%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drop-ceiling-lasers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-8850&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/pressure-class~150/pipe-material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/scale~decanewtons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brooms/bristle-material~fiberglass-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:01 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-8400&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-gauge-conversion-charts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000050036%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~467/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pressure-gauges/scale~decanewtons/ 2025-11-02 01:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8617 pages (at 51 pages/min), scraped 239 items (at 2 items/min) 2025-11-02 01:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~10-0000-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9318316703734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=brooms&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9318316703734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003424015%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:13 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-8850&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:05:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drum-dust-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity~1-1-4-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=galvanized-iron-and-steel-(class-150)-low-pressure-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-gauge-conversion-charts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~467&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=104738336527&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=buckets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B104738336527%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002150424%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~932/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~10-0000-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity~15-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity~3-1-2-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000022444%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5614T8&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225614T8%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000032995%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~2-7500/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=104738336527&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=buckets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B104738336527%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002140146%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drum-repair-patches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=104738336527&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=buckets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B104738336527%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002139065%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K461&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K461%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~932&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:47 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~10-0000-mm&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laser-levels/environment~dusty&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4177T24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224177T24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4269T33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224269T33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000082097%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5032840065813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=laser%20levels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5032840065813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055874%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~2-7500&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drum-repair-patches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity~40-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laser-squares&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000012448%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8667 pages (at 50 pages/min), scraped 243 items (at 4 items/min) 2025-11-02 01:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dry-erase-labels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-tracers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/latches/mount-type~unthreaded-plug/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000082097%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=104738336527&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=buckets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B104738336527%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099022%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4142T8&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224142T8%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-tanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=latches/mount-type~expanding-plug&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000012448%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=budgit-hoists&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-tracers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=built-in-locker-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-spring-lifts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000082097%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/latches/mount-type~expanding-plug/ 2025-11-02 01:06:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4142T8&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224142T8%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000012448%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dual-hose-reels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dual-wall-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lathe-chuck-keys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=expansion-joints/ul-file-number~mh47689&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4142T8&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224142T8%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gasket-material/for-use-with~beverage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-washer-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8718 pages (at 51 pages/min), scraped 243 items (at 0 items/min) 2025-11-02 01:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-washing-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7692142082491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=expansion%20joints&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7692142082491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dual-hose-reels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dual-wall-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992574902667&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gasket%20material&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992574902667%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967788%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bumpers/color~black-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/recessed-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct-damper-blades/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct-gaskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/workbench-tops/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7692142082491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=expansion%20joints&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7692142082491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-series~naval/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ductile-cast-iron/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bumpers/color~black&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=recessed-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-damper-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cord-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bumpers/color~black/ 2025-11-02 01:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=workbenches/steel-workbench-tops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7692142082491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=expansion%20joints&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7692142082491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/for-manufacturer-series~naval&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ductile-cast-iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8316204383934&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8316204383934%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%225064014124950%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:07:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/lavatory-equipment/for-manufacturer-series~naval/ 2025-11-02 01:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=burettes/burette-material~borosilicate-glass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrench-organizers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9054T113&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229054T113%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8767 pages (at 49 pages/min), scraped 243 items (at 0 items/min) 2025-11-02 01:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/offset-torx-drivers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~009753-25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9638090424307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=burettes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9638090424307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000037850%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/for-use-on~chutes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~012442-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~009754-25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002094703%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7432T3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227432T3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrench-organizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/cable-designation~sjt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138656%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002104428%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=offset-torx-drivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=29015K68&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2229015K68%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/for-use-on~chutes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=primers/for-use-on~sbr&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002363733%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2346K44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222346K44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~41-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~58-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=29015K69&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2229015K69%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929695%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7559498251638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=primers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7559498251638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002940377%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/for-use-with~breathable-air/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~36-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/for-use-with~fuel-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~58millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/for-use-with~hydraulic-fluid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/sae-viscosity-grade~0w-20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/gauge-size~0-022&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~breathable-air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ed23-1%252f2-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wrenches/size~58millimeters/ 2025-11-02 01:09:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~41millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8815 pages (at 48 pages/min), scraped 248 items (at 5 items/min) 2025-11-02 01:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~36mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=printers/manufacturer~sato&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070420%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964983%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~fuel-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wrenches/size~41millimeters/ 2025-11-02 01:09:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bushings/for-shaft-diameter~36mm/ 2025-11-02 01:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~hydraulic-fluid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/sae-viscosity-grade~0w-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992728463794&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=printers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992728463794%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002278025%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~ep6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7897A57&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227897A57%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/for-use-with~mild-bases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964825%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003415246%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070420%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/gauge-size~0-048&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2276K68&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222276K68%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992728463794&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=printers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992728463794%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002278025%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~ep6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/for-use-with~natural-gas-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7897A57&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227897A57%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~mild-bases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038723%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extension-cords/cable-designation~srdt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070420%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wrenches/size~ep6/ 2025-11-02 01:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179305%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19575A181&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219575A181%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992728463794&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=printers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992728463794%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002278025%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~natural-gas&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7897A57&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227897A57%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8865 pages (at 50 pages/min), scraped 250 items (at 2 items/min) 2025-11-02 01:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5121K211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225121K211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/industry-designation~srdt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/gauge-size~0-1006&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:12 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/length~10-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/reducers/for-use-with~natural-gas/ 2025-11-02 01:10:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electric-bevelers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/extension-cords/industry-designation~srdt/ 2025-11-02 01:10:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000081381%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:21 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/for-use-with~water/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001631686%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:27 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:10:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5121K211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225121K211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extension-cords/length~200-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19575A211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219575A211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electric-shears/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/gauge-size~0-148&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electric-bevelers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~water&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000030851%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5121K211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225121K211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/length~200-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=programmable-logic-controller-end-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967559%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electric-shears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19575A267&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219575A267%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~g-458-kl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~m962926-0070a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:10:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/extension-cords/length~200-ft-/ 2025-11-02 01:11:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002094089%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/length~48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8914 pages (at 49 pages/min), scraped 252 items (at 2 items/min) 2025-11-02 01:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002117967%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:08 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:11:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-contact-cleaners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-discharge-machining-(edm)/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-enclosure-paint/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001598606%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3057K52&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223057K52%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:17 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~m962926-0070a&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:11:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/gauge-size~3-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/shaft-diameter~3-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7503K23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227503K23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-contact-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000166904%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-discharge-machining-(edm)&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~p6000-eg&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-enclosure-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002271960%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=propane-tank-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2249A25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222249A25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099888%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/length~9-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/trade-name~molykote-33-extreme-low-temperature/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-tie-installation-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-tie-pullers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=propeller-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001599206%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/shaft-diameter~65-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099888%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 8961 pages (at 47 pages/min), scraped 255 items (at 3 items/min) 2025-11-02 01:12:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7469K13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227469K13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002236359%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~molykote-33-extreme-low-temperature&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~s09-010a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauging-pastes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-tie-installation-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-tie-pullers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002152061%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/trade-name~molykote-m-77-paste/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002149585%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099888%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1252K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221252K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/cable-designation~sjeow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gear-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/emergency-backup-lights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~molykote-m-77-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gimbals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361530%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002128567%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=proximity-reader-door-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7520K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227520K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=emergency-backup-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=proximity-transmitters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:00 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=layout-blues&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pry-bars/performance~nonsparking-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9012 pages (at 51 pages/min), scraped 258 items (at 3 items/min) 2025-11-02 01:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/refrigerant-recovery-cylinders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass-breakers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pry-bars/nonsparking-pry-bars-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/refrigerators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=external-torx-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=external-star-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=layout-compasses/manufacturer-model-number~c251c&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=layout-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pry-bars/nonsparking-pry-bars-6/ 2025-11-02 01:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=external-torx-socket-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=refrigerant-recovery-cylinders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/registers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil-flow-sights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7443438757064&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=layout%20compasses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7443438757064%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002104741%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=refrigerators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2124A13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222124A13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/material~cotton-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=registers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-flow-sights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/for-use-on~plastic-1/for-use-on~polystyrene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/color~brown&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ptfe-hex-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/engineers-pipe-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930351%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002940712%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/material~cotton&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9061 pages (at 49 pages/min), scraped 259 items (at 1 items/min) 2025-11-02 01:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001527274%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cable-ties/material~cotton/ 2025-11-02 01:14:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/manufacturer-model-number~02azd730g-a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil-sights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-screw-nuts/thread-size~1-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/manufacturer-model-number~02azd730g-b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8826536049195&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=calipers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8826536049195%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123450%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/erection-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=engineers%27-pipe-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9997045507261&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lead%20screw%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9997045507261%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/manufacturer-model-number~500-171-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8826536049195&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=calipers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8826536049195%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002113681%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/manufacturer-model-number~573-291-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-screw-nuts/thread-size~3-8-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-sights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1710K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221710K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8826536049195&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=calipers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8826536049195%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002149696%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8826536049195&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=calipers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8826536049195%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138333%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=erection-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9997045507261&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lead%20screw%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9997045507261%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454674%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oilers/for-use-with~oil/for-use-with~solvents/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:14:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:14:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gravity-roller-conveyors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6350K172&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226350K172%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/etching-tips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pull-chain-switch-light-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/etching-stencils/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ethernet-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9109 pages (at 48 pages/min), scraped 261 items (at 2 items/min) 2025-11-02 01:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oilers/for-use-with~oils-and-solvents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-screws/thread-size~3-8-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease-gun-extensions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=etching-tips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=etching-stencils&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/oilers/for-use-with~oils-and-solvents/ 2025-11-02 01:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/evaporator-cleaners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ethernet-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1206504422411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lead%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1206504422411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454521%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cans/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-screws/thread-size~m10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99030A988&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299030A988%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9946103073737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9946103073737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/relays/input-voltage~120v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/relays/input-voltage~36v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1206504422411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lead%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1206504422411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038594%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/relays/input-voltage~24v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=evaporator-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-screws/thread-size~m26&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7549K71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227549K71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1206504422411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lead%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1206504422411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333485%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=relays/input-voltage~36v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=relays/input-voltage~120v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9946103073737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9946103073737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=relays/input-voltage~24v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/one-way-drive-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease-removers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97048A746&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297048A746%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyebolts/thread-size~2-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7088615684115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7088615684115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002845843%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9154 pages (at 45 pages/min), scraped 263 items (at 2 items/min) 2025-11-02 01:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7088615684115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7088615684115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002849340%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7088615684115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7088615684115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001623135%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=leaf-blowers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1443843511411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=eyebolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1443843511411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452177%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7995K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227995K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9946103073737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9946103073737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=one-way-drive-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7088615684115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7088615684115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002849340%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease-removers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/relays/switching-current~140-amp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caps/pipe-size~1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/open-gear-grease/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7397459384976&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=caps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7397459384976%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000144603%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyebolts/thread-size~m20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyebolts/thread-size~8-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7088615684115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7088615684115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002849340%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-masks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/open-top-storage-bags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=relays/switching-current~140a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1443843511411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=eyebolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1443843511411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=open-gear-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:51 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1443843511411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=eyebolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1443843511411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000060837%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:16:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/relays/switching-current~140a/ 2025-11-02 01:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extension-mirrors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/orm-d-labels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=open-top-storage-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/orp-(oxidation-reduction-potential)-testers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9202 pages (at 48 pages/min), scraped 265 items (at 2 items/min) 2025-11-02 01:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/eye-wash-hose-sprayers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/replaceable-core-cam-locks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/replaceable-element-fuses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-steel-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-mirrors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=orm-d-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=orp-(oxidation-reduction-potential)-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eye-wash-hose-sprayers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=replaceable-core-cam-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/replacement-hose-for-paint-tank-sprayers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=replaceable-element-fuses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/eyeglass-storage-cabinets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fabric-reinforced-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=casters/mount-type~flange&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/face-mount-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=replacement-hose-for-paint-tank-sprayers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gripper-cylinders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/outlets/current~25-amp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyeglass-storage-cabinets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1981438481801&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=casters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1981438481801%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003011027%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=leg-dollies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=leggings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=casters/mount-type~threaded-stud&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5674K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225674K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=face-mount-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9253 pages (at 51 pages/min), scraped 265 items (at 0 items/min) 2025-11-02 01:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/current~25a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1981438481801&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=casters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1981438481801%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010668%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fast-travel-acme-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pulleys/for-use-with~rope/material~iron-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=casters/wheel-type~flat-free&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/for-use-with~rope/material~iron-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/outlets/current~25a/ 2025-11-02 01:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fast-travel-precision-acme-threaded-rods-for-precision-acme-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6460K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226460K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5674K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225674K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/respirators/compatible-with~3m-6500-series/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1981438481801&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=casters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1981438481801%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002066275%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8470423702633&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pulleys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8470423702633%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967113%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510042%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fdm-extruders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fast-travel-acme-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/for-use-with~rope/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4941T33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224941T33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3117T4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223117T4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fast-travel-precision-acme-threaded-rods-for-precision-acme-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8470423702633&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pulleys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8470423702633%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967113%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5674K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225674K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/compatible-with~3m-6500-series&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fdm-extruders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ceramic-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3211T41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223211T41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101834%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/respirators/for-use-with~odors-from-acid-gases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ceramic-rounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:00 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5834T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225834T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fan-blades/shaft-diameter~1-4-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fan-blades/shaft-diameter~5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9303 pages (at 50 pages/min), scraped 269 items (at 4 items/min) 2025-11-02 01:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/overhead-clearance-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/respirators/manufacturer-model-number~7583p100l/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8197494541975&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fan%20blades&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8197494541975%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008788%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/for-use-with~odor-from-acid-gases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8197494541975&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fan%20blades&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8197494541975%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000164864%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cga-welding-gas-regulators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=17545K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2217545K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/respirators/for-use-with~odor-from-acid-gases/ 2025-11-02 01:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5352K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225352K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levels/thread-size~m3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/overhead-doors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overhead-clearance-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/manufacturer-model-number~7583p100l&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hall-effect-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/respirators/manufacturer-model-number~815366/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/overhead-lights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4945834745524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=levels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4945834745524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fiber-optic-cleaners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108907%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:48 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levels/thread-size~m3&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overhead-doors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/overloads/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=54445T169&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2254445T169%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-benders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/manufacturer-model-number~815366&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pumps/power~1-15-hp-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overhead-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:19:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiber-optic-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002103196%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9349 pages (at 46 pages/min), scraped 272 items (at 3 items/min) 2025-11-02 01:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fiberglass-bonding-epoxies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/current~10-9-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overloads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=54445T169&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2254445T169%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9100T226&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229100T226%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:20:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001595471%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/restricted-area-signs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pumps/horsepower~1-15-hp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-drive-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fiberglass-sleeve-packing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-bonding-epoxies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4808N12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224808N12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=54445T169&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2254445T169%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oxygen-gas-labels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3556277725899&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pumps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3556277725899%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002995051%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=restricted-area-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-sleeve-packing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pumps/power~1-45-hp-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pa-speakers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oxygen-gas-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/mounting-location~bench-top&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pumps/power~1-8-hp-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lifting-nets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retaining-rings/heavy-duty-external-retaining-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002276644%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pumps/power~1-45-hp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9397 pages (at 48 pages/min), scraped 274 items (at 2 items/min) 2025-11-02 01:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6646950986674&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6646950986674%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%227528906675358%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pa-speakers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-tool-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pumps/power~1-8-hp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pumps/power~1-45-hp/ 2025-11-02 01:21:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~1-1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98585A109&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298585A109%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-wheel-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pumps/power~1-8-hp/ 2025-11-02 01:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rethreading-kits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chemours-krytox-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070707%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000128325%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pumps/power-source~vehicle-battery&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/finger-safe-terminal-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98585A109&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298585A109%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~12-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rethreading-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~12-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3556277725899&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pumps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3556277725899%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076359%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000158488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000112046%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/packing-grease already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punch-formers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=finger-safe-terminal-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98585A109&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298585A109%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/finishing-nails/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4721N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224721N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punch-stamp-and-chisel-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/retrieving-reachers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9447 pages (at 50 pages/min), scraped 274 items (at 0 items/min) 2025-11-02 01:22:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chuck-keys/chuck-key-number~t1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-base-number~2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/packing-grease already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:19 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:22:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=finishing-nails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reverse-acting-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4721N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224721N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retrieving-reachers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clamps/material~316-stainless-steel/material~ptfe/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~17-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chuck-keys/key-number~t1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/base-number~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fire-rated-door-frames/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000151071%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/chuck-keys/key-number~t1/ 2025-11-02 01:22:42 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reverse-acting-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4721N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224721N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/base-number~2/ 2025-11-02 01:22:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamps/material~316-stainless-steel-ptfe-plastic-blend&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1953K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221953K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/material~rexolite-polystyrene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~25-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~3-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:22:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/clamps/material~316-stainless-steel-ptfe-plastic-blend/ 2025-11-02 01:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-rated-door-frames&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hatch-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000080728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~3-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9491 pages (at 44 pages/min), scraped 274 items (at 0 items/min) 2025-11-02 01:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1953K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221953K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~34&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rexolite-polystyrene-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1939K18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221939K18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~3-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000050765%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/punches/for-hole-diameter~3-1-4/ 2025-11-02 01:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~3-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-base-number~bau15s&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hazardous-material-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1953K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221953K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=20445K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2220445K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~7-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pads/mounting-location~through-wall/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=headed-precision-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/punches/for-hole-diameter~3-7-8/ 2025-11-02 01:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000084360%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~54&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K237&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K237%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:46 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fixed-mount-steps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~7-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pads/mount-type~through-wall&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/hole-diameter~1-11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000126877%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/hole-diameter~1-23-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:23:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/punches/for-hole-diameter~7-32/ 2025-11-02 01:24:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pads/mount-type~through-wall/ 2025-11-02 01:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5113201929937&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5113201929937%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002279134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pail-mixers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=20375K711&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2220375K711%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/hole-diameter~1-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fixed-mount-steps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9541 pages (at 50 pages/min), scraped 277 items (at 3 items/min) 2025-11-02 01:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clips/material~aluminum-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5113201929937&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5113201929937%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247499%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flame-resistant-hats-and-hoods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5113201929937&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5113201929937%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001675450%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:24:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:24:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:24:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9975836744491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=clips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9975836744491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/hole-diameter~13-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pail-mixers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flaring-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/right-hand-threaded-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flame-resistant-hats-and-hoods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5113201929937&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5113201929937%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002246967%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/hole-diameter~25-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rigid-conduit-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/right-hand-threaded-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5113201929937&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5113201929937%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002246048%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/drive-style~phillips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flaring-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=right-hand-threaded-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rigid-duct/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rigid-conduit-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/hole-diameter~6-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=right-hand-threaded-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coatings/for-use-on~plastic-1/for-use-on~pvc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~porcelain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flat-roller-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rigid-duct&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5113201929937&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5113201929937%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001673496%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/drive-style~phillips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paint/for-use-on~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949688381693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=coatings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949688381693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929673%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9587 pages (at 46 pages/min), scraped 278 items (at 1 items/min) 2025-11-02 01:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/fastener-head-type~oval&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flexible-arm-machine-shields/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~pvc-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flat-roller-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001612946%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/coatings/for-use-on~pvc-plastic/ 2025-11-02 01:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ring-shank-nails/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint/for-use-on~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:25:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:25:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heavy-duty-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flexible-arm-machine-shields&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/paint/for-use-on~iron/ 2025-11-02 01:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heavy-duty-shelving&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flip-charts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ring-shank-nails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helical-inserts-for-plug-taps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-button-switch-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~3-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drive-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/driver-style~t-handle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flip-charts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-in-fasteners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/floor-locks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/color~red/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2653472997099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20drivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2653472997099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002308620%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-in-tapered-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/floor-marking-arrows/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9634 pages (at 47 pages/min), scraped 278 items (at 0 items/min) 2025-11-02 01:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~3-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/diameter~0-143/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/combination-wrenches/size~3-1-2/ 2025-11-02 01:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~1876&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paint-sprayer-replacement-parts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/color~red&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-marking-arrows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~1889&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002327103%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522000%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=concave-radius-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/diameter~0-143&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:37 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:26:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321499%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit/fitting-type~connector-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/painting-racks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T903&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T903%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint-sprayer-replacement-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97526A120&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297526A120%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548438%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5860541886895&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conduit&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5860541886895%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001919498%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T403&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T403%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/diameter~4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97358A104&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297358A104%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=painting-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-release-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97526A120&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297526A120%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9682 pages (at 48 pages/min), scraped 281 items (at 3 items/min) 2025-11-02 01:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pallet-mule-pallet-truck-replacement-parts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-to-close-door-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:27:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/rivet-type~hollow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~24e&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/diameter~4mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~276&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flush-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321069%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rivets/diameter~4mm/ 2025-11-02 01:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97526A120&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297526A120%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pallet-mule-pallet-truck-replacement-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002326407%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/rivet-type~solid-with-washer/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~285&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/hollow-rivets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fly-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K561&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K561%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6570T972&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226570T972%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flush-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321528%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%226049109879850%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:27:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/rivet-type~solid-with-washer&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fly-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/foam-guns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-wyes/fitting-connection~push-to-connect-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001978359%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~1-2-13&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%226049109879850%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-speed-steel-taps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9731 pages (at 49 pages/min), scraped 283 items (at 2 items/min) 2025-11-02 01:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~335&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/foam-tipped-applicators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-to-connect-tube-wyes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001978359%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324251%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%226049109879850%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/thread-size~5-8-11-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-tipped-applicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001978359%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-top-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:40 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:28:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-temperature-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/thread-size~5-8-11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:45 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fold-away-shelves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/thread-size~1-7-8-12-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rivets/thread-size~5-8-11/ 2025-11-02 01:28:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=putty/for-use-on~bronze-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/panel-mount-ball-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:28:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hilti-compatible-drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fold-away-shelves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~11-16-27&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/folding-hinges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6292651935014&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=putty&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6292651935014%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929994%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9781 pages (at 50 pages/min), scraped 284 items (at 1 items/min) 2025-11-02 01:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~1-7-8-12-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=putty/for-use-on~garolite-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~4350&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ball-joint-linkages/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panel-mount-ball-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451373%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6292651935014&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=putty&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6292651935014%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003432021%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:22 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~11-16-27&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324360%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=folding-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7605A3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227605A3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~2-3-4-4-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8351K66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228351K66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~4509&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/ball-joint-linkages-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449109%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/food-service-wrap/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/leaf-type~offset/hinge-type~piano&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~1-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%229495356926942%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hitch-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/foot-jacks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~3-8-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001973201%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969305%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8351K233&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228351K233%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000126884%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=60645K211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2260645K211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449109%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-service-wrap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1567A41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221567A41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9836 pages (at 55 pages/min), scraped 287 items (at 3 items/min) 2025-11-02 01:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foot-jacks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8351K233&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228351K233%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~585&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449109%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1567A41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221567A41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/forklift-attachments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hoist-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~5-8-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002326119%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-(schedule-80)-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8351K233&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228351K233%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T607&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T607%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-aprons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1567A41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221567A41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=forklift-attachments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:30:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99030A328&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299030A328%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/panels/shape~wire/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/robotics/robot-cell-component~gripper/manufacturer~fanuc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~7-32-20-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~m0-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:30:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:00 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:31:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/robotics/robot-cell-component~tool/for-robot-arm-manufacturer~precise-automation/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449333%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/connects-to~cord-grip/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451788%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9885 pages (at 49 pages/min), scraped 289 items (at 2 items/min) 2025-11-02 01:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panels/shape~wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/electrical-connection~wire-leads/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~796&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~m135&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~gripper/manufacturer~fanuc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6350K692&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226350K692%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9962166674953&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9962166674953%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313023%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002326738%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451614%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9985525866263&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9985525866263%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002142085%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002259996%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~tool/manufacturer~precise-automation&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/connection-type~cord-grip&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/connection-type~wire-leads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:29 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panels/shape~wire&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/freeze-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9613T132&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229613T132%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9985525866263&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9985525866263%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108937%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cords/connection-type~cord-grip/ 2025-11-02 01:31:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cords/connection-type~wire-leads/ 2025-11-02 01:31:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rod-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6968N111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226968N111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~m70&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=freeze-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fastening-tape/color~dark-blue-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-pipe-glue&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-pipe-wyes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449308%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~e211-2p&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/color~copper-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6968N111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226968N111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321708%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2355999507944&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fastening%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2355999507944%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001543160%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9935 pages (at 50 pages/min), scraped 291 items (at 2 items/min) 2025-11-02 01:32:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/parallel-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/color~dark-blue&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fuel-cans/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=76555A721&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2276555A721%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fastening-tape/color~green-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/fastening-tape/color~dark-blue/ 2025-11-02 01:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rod-ends/rod-end-type~bolt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6968N111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226968N111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~e212-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321708%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parallel-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/parker-compressed-air-filters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321220%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuel-cans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/color~green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-ends/rod-end-type~bolt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:41 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T174&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T174%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321708%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/fastening-tape/color~green/ 2025-11-02 01:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/for-use-with~carbon-dioxide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parker-compressed-air-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8204790604356&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8204790604356%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001934583%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rod-ends/shank-thread-size~1-4-20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964834%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:32:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/for-use-with~coolant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cotton-gloves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:33:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/for-use-with~ethanol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969065%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 9983 pages (at 48 pages/min), scraped 293 items (at 2 items/min) 2025-11-02 01:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~etj&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-ends/shank-thread-size~1-4-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964606%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~etj/ 2025-11-02 01:33:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~f20t12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8204790604356&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8204790604356%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000082398%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/tape-type~heat-transfer&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969065%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~f48t12-ho&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334472%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/countersinks/screw-size~no-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6154K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226154K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/material~chlorobutyl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/rod-ends/shank-thread-size~m12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rod-mounts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2355999507944&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fastening%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2355999507944%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002271560%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334457%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1501K44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221501K44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001527896%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969065%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1504K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221504K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=76455A72&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2276455A72%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/for-screw-size~no-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=53715K211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2253715K211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/rod-ends/shank-thread-size~m12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/width~3-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:33:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/countersinks/for-screw-size~no-8/ 2025-11-02 01:33:59 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvdf-push-to-connect-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-01/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=76455A72&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2276455A72%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2355999507944&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fastening%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2355999507944%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000164322%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10031 pages (at 48 pages/min), scraped 297 items (at 4 items/min) 2025-11-02 01:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/parts-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=76995A27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2276995A27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvdf-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/screw-size~no-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~fo96&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:34:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-01&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=76455A72&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2276455A72%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=983458193638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=countersinks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B983458193638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454509%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334427%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parts-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/parts-washing-tubs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001533860%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/galvanized-iron-pipe-locknuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-clamps/material~300-series-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8355K85&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228355K85%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:36 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/screw-size~no-18&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:34:37 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=faucet-cartridges/for-manufacturer~kohler&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:42 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-01&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5313593865679&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20clamps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5313593865679%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510136%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/galvanized-steel-hex-nuts-for-astm-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parts-washing-tubs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/patch-heaters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3621951154988&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=faucet%20cartridges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3621951154988%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002128308%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=galvanized-iron-pipe-locknuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5419K31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225419K31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pavement-paint/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-086/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2633K137&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222633K137%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-end-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=galvanized-steel-hex-nuts-for-astm-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10079 pages (at 48 pages/min), scraped 300 items (at 3 items/min) 2025-11-02 01:35:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pc-cable-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=patch-heaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-0935/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-099/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pavement-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-086&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~hib9005&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/garden-hose-screens/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pc-cable-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000028685%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002325921%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-0935&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-099&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-change-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gas-analyzers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~refrigerant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9613T164&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229613T164%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550513%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001533689%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=garden-hose-screens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-115/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gas-cylinder-storage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967995%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A256&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A256%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gas-dispensers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8893K136&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228893K136%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:35:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-analyzers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-115&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10125 pages (at 46 pages/min), scraped 304 items (at 4 items/min) 2025-11-02 01:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-cylinder-storage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gasket-sealed-duct/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/peel-away-coatings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/for-use-with~abrasive-materials&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/shaft-diameter~1-13-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001547922%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gate-safety-barriers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969048%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002260089%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feedthrough-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8893K146&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228893K146%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gasket-sealed-duct&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=peel-away-coatings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5706N38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225706N38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8285K24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228285K24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gate-safety-barriers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/shaft-diameter~22-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pegboard-fixtures/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-0201/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002279038%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-332/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1308/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~3-4-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8285K24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228285K24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feeler-gauges/thicknesses-included~0-001-amber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9845T55&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229845T55%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pegboard-fixtures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-0201&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10170 pages (at 45 pages/min), scraped 306 items (at 2 items/min) 2025-11-02 01:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-332&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1308&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8285K24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228285K24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~0-001-amber-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gauge-blocks/size~0-0201/ 2025-11-02 01:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539282%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9845T55&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229845T55%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~1308/ 2025-11-02 01:37:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/couplings/thread-size~3-4/ 2025-11-02 01:37:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/feeler-gauges/thickness~0-001-amber-/ 2025-11-02 01:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A199&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A199%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-1006/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-grip-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:32 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:37:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-type~unf-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1533/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-restraints&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9845T55&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229845T55%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1500t3q-cl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129835%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cpvc-tube-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~1500t3q-cl/ 2025-11-02 01:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-1006&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1533&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:37:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~1-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:37:57 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1819/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cpvc-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gauge-blocks/size~0-1006/ 2025-11-02 01:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cr2032-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~1533/ 2025-11-02 01:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/perforated-boards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10218 pages (at 48 pages/min), scraped 307 items (at 1 items/min) 2025-11-02 01:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~1-55-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1895a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~1-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1819&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hot-water-tanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quill-scales&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=perforated-boards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000146959%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~1-55-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~1819/ 2025-11-02 01:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1895a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hvac-blowers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~0-100-to-0-500-0-001-increments-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001533071%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~150-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~1895a/ 2025-11-02 01:38:40 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:38:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/feeler-gauges/thickness~0-100-to-0-500-0-001-increments-/ 2025-11-02 01:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~9-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95080A108&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295080A108%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~11-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~2357na/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~75-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~0-3-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~150mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/size~5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cryogenic-globe-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/size~9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~243/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8264408215759&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=feeler%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8264408215759%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001453797%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gauge-blocks/size~150mm/ 2025-11-02 01:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10263 pages (at 45 pages/min), scraped 308 items (at 1 items/min) 2025-11-02 01:39:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/crowfeet/size~5-8/ 2025-11-02 01:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~11-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~2357na&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/crowfeet/size~9-16/ 2025-11-02 01:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~75-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~13-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cushions/material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536899%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~2357na/ 2025-11-02 01:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~2721mf/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gauge-blocks/size~75-mm/ 2025-11-02 01:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~243&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8264408215759&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=feeler%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8264408215759%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001453797%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=raceway-outlet-box-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~18-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~243/ 2025-11-02 01:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~13-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cushions/material~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~2721mf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rack-card-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/for-use-with~v-belt-pulleys/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001547776%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~2-25-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8264408215759&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=feeler%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8264408215759%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001453797%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cushions/material~plastic/ 2025-11-02 01:39:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~2721mf/ 2025-11-02 01:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~18mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rack-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8813A82&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228813A82%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/diameter~18mm/ 2025-11-02 01:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/for-use-with~v-belt-pulleys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~2-25mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10312 pages (at 49 pages/min), scraped 309 items (at 1 items/min) 2025-11-02 01:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feet/thread-size~1-8-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutters/for-use-on~carbon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-lift-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feet/thread-size~1-12-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pfa-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967342%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/diameter~2-25mm/ 2025-11-02 01:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/racks/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9988271414601&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cutters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9988271414601%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930243%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8031429930803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=feet&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8031429930803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333464%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pfc-couplers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feet/thread-size~1-8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutters/for-use-on~tool-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~3-85-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3316A32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223316A32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pfa-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967342%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=racks/material~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/feet/thread-size~1-8-1/ 2025-11-02 01:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9988271414601&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cutters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9988271414601%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/girth-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pfc-couplers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/racks/material~brass/ 2025-11-02 01:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dampers/material~rubber-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~3-85mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967342%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/diameter~3-85mm/ 2025-11-02 01:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=girth-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iec-inlets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=radiation-detectors/measures~alpha-radiation&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~47/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dampers/material~rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10357 pages (at 45 pages/min), scraped 310 items (at 1 items/min) 2025-11-02 01:41:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=illuminators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drywall-screws/drive-style~phillips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7157463943095&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=radiation%20detectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7157463943095%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101323%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:41:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/dampers/material~rubber/ 2025-11-02 01:41:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8543T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228543T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~558/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~47&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~585/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=phillips-drywall-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diamond-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diamond-tipped-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000108547%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~47/ 2025-11-02 01:41:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-grinders/for-use-with~grinding-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~558&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fence-posts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~585&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glove-protectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~8-9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~7-8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fencing/fencing-component~floor-mount-flange/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8549892642718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=die%20grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8549892642718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964919%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~558/ 2025-11-02 01:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indexing-spring-plungers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:41:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~585/ 2025-11-02 01:41:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glycols/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10402 pages (at 45 pages/min), scraped 311 items (at 1 items/min) 2025-11-02 01:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/fabrication~extruded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glove-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~8-9mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~7-8-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fencing/component~floor-mount-flange&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rail-to-rail-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/diameter~8-9mm/ 2025-11-02 01:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001524373%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/fencing/component~floor-mount-flange/ 2025-11-02 01:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glycols&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dollies/body-material~stainless-steel-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=railcar-wheel-chocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/fabrication~extruded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~899/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=railing-bracket-bases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/specifications-met~federal-safe-drinking-water-act/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A356&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A356%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4236684941730&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dollies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4236684941730%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001574024%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002248057%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~9004ll/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/thread-size~1-1-4-8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~899&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fencing/safety-barrier-component~mounting-plate-cover&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/specifications-met~federal-safe-drinking-water-act&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~899/ 2025-11-02 01:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grating-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3565957842449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fencing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3565957842449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002052294%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114932%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~9004ll&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rails/component~top-slide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/rods/thread-size~1-2-10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~1-1-4-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=60075T93&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2260075T93%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10451 pages (at 49 pages/min), scraped 312 items (at 1 items/min) 2025-11-02 01:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~9004ll/ 2025-11-02 01:43:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:43:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9948522430991&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rails&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9948522430991%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002251944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/thread-size~1-1-4-8/ 2025-11-02 01:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3588N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223588N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grating-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fernco-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7550N111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227550N111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/thread-size~1-7-8-12-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~a1a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/rods/thread-size~1-2-10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferraz-shawmut-type-rf-fuses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:43:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:43:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3588N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223588N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease-dispensing-drum-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~1-7-8-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ferrules/fitting-connection~claw-clamp-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~a1a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/thread-size~4-40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=railway-track-aligners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rain-bibs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/thread-size~1-7-8-12/ 2025-11-02 01:43:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:43:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3588N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223588N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~a1a/ 2025-11-02 01:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease-dispensing-drum-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:43:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inserts/thread-size~m12-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease-fitting-vents/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/connection-style~claw-clamp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~4-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10495 pages (at 44 pages/min), scraped 314 items (at 2 items/min) 2025-11-02 01:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:11 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:44:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/ferrules/connection-style~claw-clamp/ 2025-11-02 01:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000122836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~12-0000-mm/diameter~12-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pin-connectors/number-of-poles~19/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/connects-to~pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inserts/thread-size~m12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pin-connectors/number-of-poles~25/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease-fitting-vents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001534818%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000077135%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inserts-for-turning-carbide-insert-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002394432%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/inserts/thread-size~m12/ 2025-11-02 01:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pin-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rat-traps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-connectors/number-of-poles~19&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/thread-size~m6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50485K856&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250485K856%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-connectors/number-of-poles~25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/tolerance-rating~undersized/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001551445%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001553546%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inserts-for-turning-carbide-insert-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/fastener-head-type~rounded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ratchet-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~m6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6134T66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226134T66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:44:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1829K332&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221829K332%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/tolerance-rating~undersized&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000148470%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/for-use-with~sulfuric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10538 pages (at 43 pages/min), scraped 317 items (at 3 items/min) 2025-11-02 01:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/head-type~rounded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000069892%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:13 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:45:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 01:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shaft-grommets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/dowel-pins/head-type~rounded/ 2025-11-02 01:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/length~0-41&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/thread-size~m10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/thread-size~m4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pinion-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001607869%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grounding-pigtails/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038594%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carbide-drill-bit-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000062738%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/for-use-on~enclosures-and-equipment&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/shaft-grommets-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50915K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250915K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insulation&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930159%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/grommets/shaft-grommets-4/ 2025-11-02 01:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pinion-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grounding-pigtails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carbide-drill-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/carbide-drill-bit-sets-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~h3-100w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:45:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/carbide-drill-bit-sets-10/ 2025-11-02 01:46:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/length~12-7-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insulation&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930159%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-point-type~notched&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~h4-100-55w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10587 pages (at 49 pages/min), scraped 318 items (at 1 items/min) 2025-11-02 01:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/carbide-drill-bits-11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001625876%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969015%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:16 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~h3-100w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/carbide-drill-bits-11/ 2025-11-02 01:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/length~17-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50715K372&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250715K372%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/h-beams/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insulation&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930159%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~h4-100-55w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~h3-100w/ 2025-11-02 01:46:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001622687%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~h4-100-55w/ 2025-11-02 01:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hair-dryers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plug-gauges/gauge-measurement-type~no-go/size-checking-gauge-class~z/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/length~2-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7195N27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227195N27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=h-beams&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:41 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:46:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-diameter~0-58/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001584650%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~1-9-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hair-dryers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/class-z-no-go-plug-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:46:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4759K55&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224759K55%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7195N27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227195N27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493511%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=razor-blade-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%22542153872595%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inventory-control-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-diameter~0-58&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10637 pages (at 50 pages/min), scraped 320 items (at 2 items/min) 2025-11-02 01:47:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~kpr12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~12-0-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=23065A503&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2223065A503%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-018/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000095065%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7195N27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227195N27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002892544%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-020/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=60135K91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2260135K91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~kpr12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~15-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/length~47-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-018&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:47:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:47:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~kpr12/ 2025-11-02 01:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509201%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-diameter~10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001585313%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/length~5-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001527241%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hand-torches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-02&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50815K104&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250815K104%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001630200%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:54 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-018&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jacks/application~data/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~29-0-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-diameter~10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001519361%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4746K24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224746K24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-072/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-torches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/handbooks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamer-sets/blade-shape~straight/size-adjustability~adjustable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003250388%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10686 pages (at 49 pages/min), scraped 323 items (at 3 items/min) 2025-11-02 01:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000112915%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jacks/application~data-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7218065360791&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamer%20sets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7218065360791%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038768%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003681173%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2478K113&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222478K113%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-072&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/handrail-bracket-bases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/jacks/application~data-1/ 2025-11-02 01:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handbooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:26 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamer-sets/blade-shape~straight/size-adjustability~adjustable&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/taper-pin-number~9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/for-use-on~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001530959%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~5-8-11-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~11-7-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~11-9-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:39 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-072&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handrail-bracket-bases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=940728682702&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=jam%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B940728682702%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333424%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-102/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/for-taper-pin-number~9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jewelers-saw-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~60/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/for-use-on~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:51 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~5-8-11-2&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~11-7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~11-9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reamers/for-taper-pin-number~9/ 2025-11-02 01:48:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:48:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reamers/for-use-on~aluminum/ 2025-11-02 01:49:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~13-3-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000043753%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hardware-for-toilet-and-urinal-partitions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000152274%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-102&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~60-ga-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10731 pages (at 45 pages/min), scraped 325 items (at 2 items/min) 2025-11-02 01:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/harsh-environment-butt-splices/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=job-ticket-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2281T53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222281T53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2276T13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222276T13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001557525%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/drill-bit-size~60-ga-/ 2025-11-02 01:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~13-3-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=joints/connects-to~hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hardware-for-toilet-and-urinal-partitions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~17-5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:25 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-102&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000043935%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hats/clothing-size~l/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=harsh-environment-butt-splices&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992188250120&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=joints&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992188250120%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002393955%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2281T53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222281T53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-1575/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~19-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T491&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T491%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hazardous-location-air-conditioners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~17-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hats/size~l&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2281T53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222281T53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~10-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-1575&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~19-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000051559%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T491&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T491%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hats/size~l/ 2025-11-02 01:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fhs-440-6-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:49:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hazardous-location-air-conditioners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001545661%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000068308%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2245T4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222245T4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10781 pages (at 50 pages/min), scraped 326 items (at 1 items/min) 2025-11-02 01:50:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~10-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A587&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A587%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12975N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212975N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T491&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T491%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~12-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~1-1-w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~24-7-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001633181%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:50:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~13-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001492268%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiber-optic-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:30 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~10-5-8&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-221/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A587&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A587%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:33 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:50:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~12-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K257&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K257%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~24-7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001607030%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000083199%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-221&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~29-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0760&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~13-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0770&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A587&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A587%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2281T67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222281T67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:50:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001551726%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000153258%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000065096%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001618159%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiber-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~500k&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:06 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-221&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10832 pages (at 51 pages/min), scraped 329 items (at 3 items/min) 2025-11-02 01:51:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~29-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:10 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0770&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:51:10 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0760&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:51:11 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400947%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2281T67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222281T67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~29-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heat-pipe-bending-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/125-watt-light-bulbs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000089483%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-236/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~14-3-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7030K64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227030K64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0860&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2281T67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222281T67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~29-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-pipe-bending-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-236&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000128671%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fiberglass-rounds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~14-3w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000060163%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~32-3-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000089483%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001538023%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:50 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0860&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~14-3w/ 2025-11-02 01:51:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T156&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T156%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~15-7-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-(frp)-rounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~2-9-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~32-3-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000089483%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10879 pages (at 47 pages/min), scraped 331 items (at 2 items/min) 2025-11-02 01:52:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~2-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~38-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000036901%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001593666%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~17-w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~15-7w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~26-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~2-9-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T688&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T688%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:52:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:52:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001490349%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~15-7w/ 2025-11-02 01:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~38-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001589825%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heavy-duty-cable-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heavy-duty-piano-hinges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heavy-pattern-thimbles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1732N281&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221732N281%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~3-3-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~26mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000018597%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/length~26mm/ 2025-11-02 01:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~2-5-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2565T75&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222565T75%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heavy-duty-cable-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heavy-duty-piano-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heavy-pattern-thimbles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knurled-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~3-3-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~21-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10926 pages (at 47 pages/min), scraped 334 items (at 3 items/min) 2025-11-02 01:53:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001606361%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-pans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~2-5w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helical-gear-racks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/end-type~toggle/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~2-5w/ 2025-11-02 01:53:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~21-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~2-w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~21-1-w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~46-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-sinks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001546791%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helical-gear-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001491314%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/end-type~toggle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001491221%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1365&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K247&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K247%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~6-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001945191%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K236&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K236%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~46mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~3-1-w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/insulation-hanger-studs-5/hanger-stud-type~slot/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000137223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001546791%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:53:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/length~46mm/ 2025-11-02 01:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001492423%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:01 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1365&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:54:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/30-watt-light-bulbs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~no-34/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 10971 pages (at 45 pages/min), scraped 336 items (at 2 items/min) 2025-11-02 01:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pin-terminals/insulation~noninsulated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~6-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T038&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T038%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/insulation-hanger-studs-5/hanger-stud-type~slot&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001546791%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001617402%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lanyard-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/material~plastic-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~no-34&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/noninsulated-pin-terminals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~30w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:30 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~400-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~no-34/ 2025-11-02 01:54:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:54:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pins/noninsulated-pin-terminals/ 2025-11-02 01:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/retractable-spring-plungers/fastener-head-type~ring/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1835&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~30w/ 2025-11-02 01:54:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-collar-lock-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/fastener-head-type~rounded/shaft-type~barbed/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:48 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 500 Internal Server Error 2025-11-02 01:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:48 [scrapy.spidermiddlewares.httperror] (PID: 148) INFO: Ignoring response <500 https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true>: HTTP status code is not handled or not allowed 2025-11-02 01:54:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-spot-paste/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~400w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/pull-ring-retractable-spring-plungers-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/number-of-pieces~5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:54:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-collar-lock-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~7-3-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lathes/cut-type~chamfering&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/round-head-barbed-dowel-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~400w/ 2025-11-02 01:55:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pins/pull-ring-retractable-spring-plungers-7/ 2025-11-02 01:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001547342%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11019 pages (at 48 pages/min), scraped 337 items (at 1 items/min) 2025-11-02 01:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2919599794209&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lathes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2919599794209%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002256108%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-spot-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pins/round-head-barbed-dowel-pins/ 2025-11-02 01:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~7-5-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~7-3w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-temperature-spray-paint/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~7-3w/ 2025-11-02 01:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-traction-overshoes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~7-5w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2160&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-temperature-spray-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead/material~lead&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~7-5w/ 2025-11-02 01:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-vacuum-film/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:46 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:55:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000067508%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2188&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-traction-overshoes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8678154003786&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lead&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8678154003786%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511250%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:57 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2160&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/highlighting-markers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~130-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000114024%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-vacuum-film&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-taper-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~143-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~141-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11067 pages (at 48 pages/min), scraped 337 items (at 0 items/min) 2025-11-02 01:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000102687%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/corner-padlock-hasps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000112811%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000153325%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=highlighting-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000114024%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pintle-hitches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/threaded-taper-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2435&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:56:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:56:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/extra-reach-padlockable-corner-hasps-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pins/threaded-taper-pins/ 2025-11-02 01:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-bases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000092653%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lighted-crimpers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hinges/extra-reach-padlockable-corner-hasps-9/ 2025-11-02 01:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000114024%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pintle-hitches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~339-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-drain-cleaner-heads/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lime-stain-removers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~34-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-elbows/fitting-connection~barbed-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000068359%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000092653%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-bases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:00 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000068305%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11116 pages (at 49 pages/min), scraped 337 items (at 0 items/min) 2025-11-02 01:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-drain-cleaner-heads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/linear-bearing-housings/housing-type~flange-mount-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levels/thread-size~1-4-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/honeycomb-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-elbows/connection-style~barbed&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hook-and-jaw-turnbuckles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000092653%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filter-bag-housings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fasteners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4945834745524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=levels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4945834745524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pipe-elbows/connection-style~barbed/ 2025-11-02 01:57:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=linear-bearings/housing-type~flange-mount/linear-bearing-component~housing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hook-and-jaw-turnbuckles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:57:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2910&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3403647620739&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=linear%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3403647620739%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002064158%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/body-shape~hex/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/finish~epoxy-coated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fasteners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/for-use-with~acetone/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000010143%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fitting-filters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filter-paper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:54 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2910&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/body-shape~hex&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/linemans-boots/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/finish~epoxy-coated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lined-tube-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~acetone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055981%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11161 pages (at 45 pages/min), scraped 337 items (at 0 items/min) 2025-11-02 01:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/round-head-square-neck-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001546373%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fitting-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~66-66-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/for-use-with~chlorine/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965604%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lineman%27s-boots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000159147%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055981%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/for-use-with~liquid-methane/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-cutting-shears/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-head-square-neck-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:31 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~66-66-mm&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:58:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~9-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~chlorine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/for-use-with~liquid-nitrogen/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:42 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 01:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000025136%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965544%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055981%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~liquid-methane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=liquid-level-transmitters/liquid-level-sensor-type~float&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-cutting-shears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/for-use-with~ethylene-glycol-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~liquid-nitrogen&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965075%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:58:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1299699905441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=liquid%20level%20transmitters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1299699905441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003266252%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3438&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hot-water-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968988%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964862%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:06 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 01:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/pipe-size~1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11208 pages (at 47 pages/min), scraped 337 items (at 0 items/min) 2025-11-02 01:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3441N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223441N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3966N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223966N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000127813%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hot-water-storage-tanks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98805K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298805K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hot-water-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rtd-thermometers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/pipe-size~1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3520&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/pipe-size~3-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3966N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223966N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hot-water-storage-tanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/pipe-size~5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000033831%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000054679%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3535&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rub-rails/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rtd-thermometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:45 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3520&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/pipe-size~3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3966N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223966N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000133564%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/pipe-size~5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/shape~180-bend/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119922%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 01:59:59 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3535&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 01:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=littelfuse-type-fls-r-fuses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rub-rails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hvlp-(high-volume-low-pressure)-spray-guns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000046129%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:00:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber/material~rubber-1/material~recycled-rubber-over-buna-n-foam/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11255 pages (at 47 pages/min), scraped 339 items (at 2 items/min) 2025-11-02 02:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/for-use-with~sodium-nitrate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hydraulic-cranes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/shape~180-bend&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hydraulic-crimpers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968311%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hvlp-(high-volume-low-pressure)-spray-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/thread-type~npt/pipe-size~1-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310952%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/material~rubber/material~recycled-rubber-over-buna-n-foam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=linear-air-actuators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98805K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298805K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=linear-ball-guides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-cranes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-crimpers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511637%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=44605K563&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2244605K563%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/thread-type~npt/pipe-size~1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber/shape~round/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=67985T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2267985T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98805K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298805K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/thread-type~npt/pipe-size~2-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000156884%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shovels/environment~sanitary/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lock-cylinders-for-cam-and-drawer-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/shape~round&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4055&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4170&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98805K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298805K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11305 pages (at 50 pages/min), scraped 341 items (at 2 items/min) 2025-11-02 02:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/thread-type~npt/pipe-size~2-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:11 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000058017%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000041793%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hygienic-shovels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000101615%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-carabiners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-legs/mount-type~hammer-on/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/identification-badge-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:22 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4055&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000104826%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:24 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4170&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:30 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:01:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000058017%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filters/hvac-filter-trade-size~14-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-guards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-feet/mount-type~hammer-in&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-hammers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=identification-badge-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ignition-wrench-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-tongs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rubber-feet/mount-type~hammer-in/ 2025-11-02 02:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/impact-indicators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:01:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:01:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000058017%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/trade-size~14-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-guards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filters/hvac-filter-trade-size~18-30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/filters/trade-size~14-24/ 2025-11-02 02:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ignition-wrench-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dock-lights/light-technology~incandescent/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-flange-hole-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=impact-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11350 pages (at 45 pages/min), scraped 341 items (at 0 items/min) 2025-11-02 02:02:08 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/indexing-balls/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filters/filter-bag-trade-size~8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/trade-size~18-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-hangers/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=incandescent-dock-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-flange-hole-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/filters/trade-size~18-30/ 2025-11-02 02:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-insulation-elbows/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lockout-tagout-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/trade-size~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=finger-brake-tape-measures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indexing-balls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-sheets/color~yellow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-hangers/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/filters/trade-size~8/ 2025-11-02 02:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-insulation-elbows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-arms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pipe-hangers/material~aluminum/ 2025-11-02 02:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-sheets/color~yellow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4830&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=finger-ring-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4855&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-v-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000095874%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11399 pages (at 49 pages/min), scraped 341 items (at 0 items/min) 2025-11-02 02:03:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=finger-touch-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000022449%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4965&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4830&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:21 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4855&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000115748%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rulers/length~10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inspection-area-signs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locksmiths-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-40466&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-5620&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:34 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4965&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:03:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-82093&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inspection-area-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locksmiths%27-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-detectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-gasket-makers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001625651%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-5520&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-5580&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-extinguisher-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11450 pages (at 51 pages/min), scraped 341 items (at 0 items/min) 2025-11-02 02:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation-batts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-5630&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000030039%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation-paint/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000085343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-thread-reducers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=long-nose-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000093094%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:19 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-5520&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:04:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-prism-primers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/interchangeable-drives/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation-batts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:29 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-5630&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:04:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:04:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-thread-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000085343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-thread-reducers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loops/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=interchangeable-drives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-6875&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=low-temperature-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-thread-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rulon-j-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000085343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/piston-ring-expanders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loops/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038058%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loops/mount-type~weld-on&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/loops/material~aluminum/ 2025-11-02 02:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11496 pages (at 46 pages/min), scraped 341 items (at 0 items/min) 2025-11-02 02:05:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982395029765&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=loops&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982395029765%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010804%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulon-j-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:12 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:05:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=piston-ring-expanders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-resistant-shirts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038058%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-machining-type~tapping&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-machining-type~turning&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pistons/system-of-measurement~metric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002139105%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002131244%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pitchforks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/isopropyl-alcohol/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-use-on~ball-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038058%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pistons/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/j-lock-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929731%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-use-on~wire-rope&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=low-flow-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002139105%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pitchforks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981729408846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981729408846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=isopropyl-alcohol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930483%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:05:57 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:06:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rust-inhibiting-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=j-lock-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/j-style-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11544 pages (at 48 pages/min), scraped 341 items (at 0 items/min) 2025-11-02 02:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/moisture-resistant-grease/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rust-inhibitors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002139105%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rw42-batteries/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rust-inhibiting-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-stop-putty&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=low-profile-casters-(capacity-up-to-700-lbs)&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=j-style-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/moisture-resistant-grease-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rust-inhibitors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rw42-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~21-5000-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%226310464163376%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plank-decking/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-ball-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-bar-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000079937%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8708T165&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228708T165%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jobsite-fans/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-bulk-dump-trucks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:54 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~21-5000-mm&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/taper-pin-number~2-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lps-tapmatic-cutting-fluids&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plank-decking&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-ball-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-bar-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=low-profile-rivets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8708T165&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228708T165%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11591 pages (at 47 pages/min), scraped 341 items (at 0 items/min) 2025-11-02 02:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jobsite-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-bulk-dump-trucks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/taper-pin-number~2-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sae-50-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-butterfly-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/k-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002218963%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8708T165&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228708T165%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/taper-pin-number~7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sae-50-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000158698%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-butterfly-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fixed-height-workbench-legs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=receiver-tanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=k-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002218963%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:07:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:07:44 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/taper-pin-number~7&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:07:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:07:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:07:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:07:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/for-machining-type~electrical-discharge-machining/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-crates/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keyless-locking-devices/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keypad-locksets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flags/u-s-state-territory~alaska/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002218963%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=receptacle-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-use-on~vacuum-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machinery-movers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11638 pages (at 47 pages/min), scraped 341 items (at 0 items/min) 2025-11-02 02:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-machining-type~edm-electrical-discharge-machining-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-crates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929724%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keyless-locking-devices&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keypad-locksets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/lubricants/for-machining-type~edm-electrical-discharge-machining-/ 2025-11-02 02:08:21 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:08:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flags/state-us-territory~alaska&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2158K34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222158K34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/flags/state-us-territory~alaska/ 2025-11-02 02:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=recirculating-chillers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease/trade-name~molykote-112-high-performance/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reclosable-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=maintained-push-button-pilot-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnifying-lenses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flags/u-s-state-territory~iowa&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-knobs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rectifiers/electrical-component-trade-number~gbj2502&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~delvac-1230&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/safety-wire-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/moisture-resistant-seal-and-o-ring-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-manifold-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9755626796405&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flags&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9755626796405%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000153174%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9469999789402&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rectifiers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9469999789402%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005370461%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002085602%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-mesh-screens/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%225544510310534%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7545T83&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227545T83%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11687 pages (at 49 pages/min), scraped 342 items (at 1 items/min) 2025-11-02 02:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7615N38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227615N38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-wire-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flame-resistant-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/salt-water-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1204K42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221204K42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-manifold-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002085602%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-mesh-screens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sample-vials/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1204K42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221204K42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~mobil-dte-extra-heavy&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/fitting-connection~interlocking-bracket/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=salt-water-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flammable-liquids-containers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002078838%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/fitting-connection~quick-disconnect-clamp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~mobilgear-600-xp-220&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002085602%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sample-vials&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1204K42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221204K42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2158K37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222158K37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/fitting-connection~quick-clamp-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/connection-type~interlocking-bracket&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081677%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/laminated-steel-shim-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361293%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reducers/connection-type~interlocking-bracket/ 2025-11-02 02:10:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2158K86&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222158K86%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/connection-type~quick-disconnect-clamp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11739 pages (at 52 pages/min), scraped 345 items (at 3 items/min) 2025-11-02 02:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sanding-rolls/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-s-hooks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-scoops/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reducers/connection-type~quick-disconnect-clamp/ 2025-11-02 02:10:16 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laminated-steel-shim-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~boric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sandpaper/grit~2000/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sandpaper/grit~1800/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanding-rolls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964962%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanged-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-s-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-scoops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mechanical-interlocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=53505K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2253505K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sandpaper/grit~2000-grit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sandpaper/grit~1800-grit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubrication-brushes/handle-material~metal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sandpaper-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sandpaper/grit~2000-grit/ 2025-11-02 02:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sanitary-barbed-tube-fittings/fitting-type~connector-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=media-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:10:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sandpaper/grit~1800-grit/ 2025-11-02 02:11:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6426418044677&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubrication%20brushes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6426418044677%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101654%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh-buckets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh-doors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11785 pages (at 46 pages/min), scraped 347 items (at 2 items/min) 2025-11-02 02:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/shape~manifold-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sandpaper-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7237T81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227237T81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lumber-crayons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-barbed-tube-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lumber-marking-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/performance~moisture-resistant-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-doors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/shape~manifold&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-lights/environment~sanitary/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lute-rakes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/thickness~0-04/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reducers/shape~manifold/ 2025-11-02 02:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/performance-properties~moisture-resistant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=m16-glands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/flange-od~4-21-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/thickness~0-094/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-doors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plastic-sheets/performance-properties~moisture-resistant/ 2025-11-02 02:11:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-machine-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-detectable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000071958%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~0-04&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-component-knobs/material~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lead-pencils/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:11:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sanitary-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1747N455&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221747N455%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001457391%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~0-094&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sanitary/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11830 pages (at 45 pages/min), scraped 348 items (at 1 items/min) 2025-11-02 02:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001458394%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-component-knobs/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-pencils&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/thread-size~m2-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-wastebaskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/machine-component-knobs/material~iron/ 2025-11-02 02:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metals/fabrication~extruded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/thickness~7-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screws/military-specification~ms35206-201/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flanges/material~bronze-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1062337176277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=metals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1062337176277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002248057%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/for-use-with~water&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-signs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-wastebaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/material~steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967559%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~7-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-screws/military-specifications-met~ms35206-201&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/material~bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducing-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screws/military-specification~ms51958-84/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001465187%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/machine-screws/military-specifications-met~ms35206-201/ 2025-11-02 02:13:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/flanges/material~bronze/ 2025-11-02 02:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11881 pages (at 51 pages/min), scraped 349 items (at 1 items/min) 2025-11-02 02:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screws/military-specification~ms51957-68/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8492K162&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228492K162%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducing-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-screws/military-specifications-met~ms51958-84&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screws/military-specification~ms51959-83/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-flat-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/thread-type~nptf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/machine-screws/military-specifications-met~ms51958-84/ 2025-11-02 02:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-screws/specifications-met~ms51957-68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8492K162&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228492K162%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges-for-sodium-hydroxide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-screws/specifications-met~ms51959-83&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/machine-screws/specifications-met~ms51957-68/ 2025-11-02 02:13:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/life-preservers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-socket-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/machine-screws/specifications-met~ms51959-83/ 2025-11-02 02:13:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8492K162&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228492K162%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lifting-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:13:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=refractory-ceramic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=life-preservers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=made-to-order-reusable-metal-panel-air-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11931 pages (at 50 pages/min), scraped 349 items (at 0 items/min) 2025-11-02 02:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:09 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lifting-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=refrigerator-doors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=refrigerator-thermometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=micro-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=register-diverters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lightning-conductors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-clasps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:35 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastics/performance~high-strength-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastics/system-of-measurement~metric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=regulator-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plate-lifters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-core-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lightning-conductors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lint-free-swabs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=milling-machine-vises&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastics/performance-properties~high-strength&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastics/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=regulators/for-use-with~methane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plate-lifters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lint-free-swabs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plastics/performance-properties~high-strength/ 2025-11-02 02:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 11977 pages (at 46 pages/min), scraped 349 items (at 0 items/min) 2025-11-02 02:15:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=778212328374&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B778212328374%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7367171817318&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=regulators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7367171817318%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965661%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-sweepers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=regulators/for-use-with~sulfuric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/scrapers/blade-shape~half-round/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flat-shoulder-rod-end-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7897A35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227897A35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:15:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7367171817318&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=regulators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7367171817318%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screen-mounts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locking-brackets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnets/color~fluorescent-orange&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scrapers/blade-shape~half-round&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=regulators/outlet-pipe-size~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/fitting-connection~flanged/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flexible-bit-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3815738403502&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=magnets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3815738403502%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002193031%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3182438425377&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3182438425377%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000031292%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7367171817318&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=regulators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7367171817318%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454159%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screen-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=motor-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1120N34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221120N34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=47345K18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2247345K18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-30516/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/connection-style~flanged&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=motor-isolators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-epoxy-adhesives/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plugs/connection-style~flanged/ 2025-11-02 02:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12028 pages (at 51 pages/min), scraped 351 items (at 2 items/min) 2025-11-02 02:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/for-use-with~hydraulic-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=47345K18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2247345K18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flexible-flashlights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-30516&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=relays/switching-current~120-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mountable-shaft-collars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-epoxy-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7088615684115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=relays&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7088615684115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001625827%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manhole-cover-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=relays/switching-current~3-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/for-use-with~hydraulic-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=47345K18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2247345K18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/manifolds/fitting-connection~loc-line/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:16:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6564K99&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226564K99%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7088615684115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=relays&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7088615684115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001631756%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/manifolds/fitting-connection~quick-clamp-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964432%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flexible-pvc-pipe-reducing-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/coupling-component~plug/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdriver-power-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3507K112&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223507K112%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manifolds/connection-style~loc-line&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/power-source~manual/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manifolds/connection-style~quick-clamp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricant-powders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/plug-type~end&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/manifolds/connection-style~loc-line/ 2025-11-02 02:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdriver-power-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=releasable-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12076 pages (at 48 pages/min), scraped 353 items (at 2 items/min) 2025-11-02 02:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/manifolds/connection-style~quick-clamp/ 2025-11-02 02:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cap-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manifolds/thread-type~bspt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plugs/plug-type~end/ 2025-11-02 02:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/power-source~manual&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8489049994451&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=manifolds&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8489049994451%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000140314%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricant-powders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flip-chart-paper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/lubricant-type~mold-release/additive~silicone/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076424%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=remote-grease-filler-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/type~cap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manual-motor-starters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=multiple-diameter-cord-grips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plugs/type~cap/ 2025-11-02 02:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/lubricant-type~mold-release-lubricant/additive~silicone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=multiports&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=multipurpose-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plunge-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manway-gaskets/material~fluoroelastomer&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001533324%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floating-joints&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=map-tubes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=maple-top-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=removable-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8728477131064&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=manway%20gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8728477131064%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001549198%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12127 pages (at 51 pages/min), scraped 353 items (at 0 items/min) 2025-11-02 02:18:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:18:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plunge-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:14 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manway-gaskets/material~fluoroelastomer&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/m-clips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plungers/thread-size~8-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pneumatic-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=marine-epoxy-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pneumatic-butterfly-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~no-0/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pneumatic-isolation-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=network-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plungers/thread-size~8-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=m-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/fitting-connection~push-to-connect-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pneumatic-mixers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-butterfly-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9219809646750&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9219809646750%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000060837%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~no-0&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-isolation-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-base-trouble-lights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/connection-style~push-to-connect&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:18:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/screwdrivers/size~no-0/ 2025-11-02 02:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-mixers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirator-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nipples/connection-style~push-to-connect/ 2025-11-02 02:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12169 pages (at 42 pages/min), scraped 353 items (at 0 items/min) 2025-11-02 02:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-particle-testers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-pole-detectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-base-trouble-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/thread-type~ght&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-pole-viewing-paper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/podiums/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pointed-tip-retrievers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=masks/for-use-with~solid-particles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000079141%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-particle-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-pole-detectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-pole-viewing-paper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3028195520048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=masks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3028195520048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967751%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=podiums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pointed-tip-retrievers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mast-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nozzles/for-use-with~hydraulic-fluid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/for-use-with~sulfuric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/for-use-with~organic-vapors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/spray-pattern~stream&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/manufacturer-model-number~6006&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179325%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/for-use-with~hydraulic-fluid-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/manufacturer-model-number~7506n95&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7846A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227846A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000076981%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12221 pages (at 52 pages/min), scraped 353 items (at 0 items/min) 2025-11-02 02:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nsf%252fansi-standard-compression-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002094405%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/manufacturer-model-number~815178&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nozzles/for-use-with~hydraulic-fluid-1/ 2025-11-02 02:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nsf-ansi-standard-sinks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002124083%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5834T86&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225834T86%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/number-jobbers-drill-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002104189%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=54445T189&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2254445T189%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/maple-top-tables/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/thread-size~2-1-2-4-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7846A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227846A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polishing-wax/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polybutylene-manifolds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=number-jobbers%27-drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002104189%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/niosh-approval~tc-84a-0901&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~2-1-2-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7846A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227846A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polishing-wax&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/respirators/niosh-approval~tc-84a-0901/ 2025-11-02 02:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/niosh-approval-number~tc-23c-0065&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/screws/thread-size~2-1-2-4/ 2025-11-02 02:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hollow-rods/material~polycarbonate/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polybutylene-manifolds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polycarbonate-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002104189%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/thread-size~4-48/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002063183%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-controls/for-use-with~dry-material&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12268 pages (at 47 pages/min), scraped 355 items (at 2 items/min) 2025-11-02 02:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/thread-size~5-40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9072876622555&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20controls&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9072876622555%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969082%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polycarbonate-hollow-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polycarbonate-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:19 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~4-48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002063183%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~10-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8153K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228153K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=meeting-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334791%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polycarbonate-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~5-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000139805%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retainers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002063183%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334642%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:46 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~10-32&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retaining-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/thread-size~9-16-20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/nut-type~locknut/thread-size~m12-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polycarbonate-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heat-elements/for-manufacturer~master-appliance/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mat-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=membrane-sealants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mauser-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12313 pages (at 45 pages/min), scraped 356 items (at 1 items/min) 2025-11-02 02:22:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyester-felt-filters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~9-16-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~m12-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=master-appliance-compatible-heating-elements&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449390%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mat-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000009660048%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-4-28-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mauser-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh/mesh-size~121-121&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyester-felt-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh/mesh-size~13-13&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334548%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3985461849490&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mesh&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3985461849490%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002056311%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/tip-type~cup/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mesh/mesh-size~3-5-3-5-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3985461849490&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mesh&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3985461849490%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002056903%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9275T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229275T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh/mesh-size~165-165&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyethylene-film/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9318T27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229318T27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/tip-type~cup&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3985461849490&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mesh&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3985461849490%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002053856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh/mesh-size~3-5-3-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002241134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12362 pages (at 49 pages/min), scraped 358 items (at 2 items/min) 2025-11-02 02:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000011539%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~7-8-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/mesh/mesh-size~3-5-3-5/ 2025-11-02 02:23:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-film&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-switches/pipe-size~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/color~copper-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyethylene-plastic-squares/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333387%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8779931053344&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8779931053344%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000158851%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nw25-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002241134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-transmitters/pipe-size~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/color~copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-plastic-squares&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retrieving-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flow-valves/valve-type~ball/fitting-connection~socket-connect-union/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/formulation~lacquer/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh-fencing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5845522290168&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20transmitters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5845522290168%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000115106%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-slides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sealants/color~copper/ 2025-11-02 02:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002241134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyethylene-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-valves/valve-type~ball/connection-style~socket-connect-union&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/formulation~lacquer&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-bulk-storage-crates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-ring-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12414 pages (at 52 pages/min), scraped 358 items (at 0 items/min) 2025-11-02 02:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=339191585183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B339191585183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247047%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:08 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002090201%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-detectable-hair-nets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~118&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:16 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-roller-chain-guards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-filled-epoxies/manufacturer~devcon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000152852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~252/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polypropylene-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7220508746168&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=metal%20filled%20epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7220508746168%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002263064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/for-use-with~acetic-acid-white-vinegar&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74575A83&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274575A83%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ribbed-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/for-use-with~isopropyl-alcohol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/for-use-with~methyl-ethyl-ketone-mek&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~252&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984513%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~292i&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965701%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002134874%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-ball-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polypropylene-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~3525/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sealants/manufacturer-model-number~292i/ 2025-11-02 02:25:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~403/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6937T6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226937T6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12463 pages (at 49 pages/min), scraped 359 items (at 1 items/min) 2025-11-02 02:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~420/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/relief-valves/body-material~polysulfone/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~3525&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ridgid-vf6000-vacuum-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=offset-keys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~403&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110202%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyurethane-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~420&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002080139%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polysulfone-relief-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=right-angle-gear-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74545A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274545A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~50172/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002093679%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74765A53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274765A53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fluorescent-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyurethane-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=45855K102&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2245855K102%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74545A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274545A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyurethane-rounds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~50172&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-drum-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-embedded-thrust-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12512 pages (at 49 pages/min), scraped 360 items (at 1 items/min) 2025-11-02 02:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~571/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126379%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=45855K102&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2245855K102%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74765A53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274765A53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~574/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74545A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274545A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyurethane-rounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-try-squares&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7605A7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227605A7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~571&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=45855K102&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2245855K102%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74765A53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274765A53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~574&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ring-gauges/pipe-size~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:35 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002096053%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7605A7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227605A7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flux/military-specification~mil-s-6872/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092765%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~8265-s/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5131635916106&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ring%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5131635916106%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000074371%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1810A53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221810A53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=on%2Foff-buttons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:26:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:26:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:26:53 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~574&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:26:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2121A143&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222121A143%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:26:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/on%2Foff-buttons/ 2025-11-02 02:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7605A7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227605A7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flux/military-specifications-met~mil-s-6872&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/portable-temperature-controllers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~8265-s&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12561 pages (at 49 pages/min), scraped 362 items (at 2 items/min) 2025-11-02 02:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/flux/military-specifications-met~mil-s-6872/ 2025-11-02 02:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133797%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=micrometers/manufacturer-model-number~t436-1xfl-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=micropipette-tips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ring-links&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=portable-temperature-controllers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7605A11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227605A11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5207874411218&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=micrometers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5207874411218%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126316%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlet-strips/certification~ul-listed&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ring-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7687249081189&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlet%20strips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7687249081189%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002091847%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-packing-dunnage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~qfs16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~rtv108/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/current-voltage~0-2-amp-115v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/shape~rectangle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/voltage~120v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/sealants/manufacturer-model-number~pc-7227/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978812115307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978812115307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001993542%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978812115307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978812115307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002309280%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978812115307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978812115307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553758%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mig-gun-contact-tips/manufacturer-model-number~magnum-300&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~qfs16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~rtv108&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3741T33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223741T33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12608 pages (at 47 pages/min), scraped 363 items (at 1 items/min) 2025-11-02 02:28:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oval-compression-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6161877157496&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mig%20gun%20contact%20tips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6161877157496%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324029%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002127296%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/post-shackle-padlocks already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002151984%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978812115307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978812115307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002309280%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/sealants/manufacturer-model-number~pc-7227/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7545A471&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227545A471%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-clamping-hangers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6161877157496&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mig%20gun%20contact%20tips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6161877157496%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324029%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivet-nuts/thread-size~3-8-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivet-nuts/thread-size~5-16-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978812115307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978812115307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002309280%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealers/for-use-on~linoleum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/post-shackle-padlocks already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7545A471&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227545A471%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7862604308299&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivet%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7862604308299%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7862604308299&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivet%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7862604308299%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333458%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:41 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 02:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 02:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing/plastic-clamping-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6161877157496&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mig%20gun%20contact%20tips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6161877157496%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324029%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/for-use-on~linoleum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971113029698&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=framing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971113029698%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%222379302435520%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7545A471&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227545A471%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealers/for-use-on~plastic-1/for-use-on~polyethylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:57 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 02:28:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7862604308299&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivet%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7862604308299%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:28:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 02:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foil-tape/material~lead&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984350057510&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984350057510%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930472%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3166T41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223166T41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:29:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12656 pages (at 48 pages/min), scraped 364 items (at 1 items/min) 2025-11-02 02:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8825112728704&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=foil%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8825112728704%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511250%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mig-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/fastener-head-type~flat/material~110-copper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7108T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227108T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:13 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:29:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/for-use-on~polyethylene-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7862604308299&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivet%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7862604308299%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=folding-benches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sealers/for-use-on~polyethylene-plastic/ 2025-11-02 02:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8825112728704&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=foil%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8825112728704%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511250%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealing-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/copper-flat-head-solid-rivets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=folding-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=folding-chair-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:34 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealing-cords/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gang-boxes/outlet-box-designation~1-gang&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%224465386799419%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8825112728704&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=foil%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8825112728704%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511250%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealing-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/diameter~0-147&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5301691116915&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gang%20boxes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5301691116915%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003015060%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97440A325&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297440A325%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=garolite-angle-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/diameter~1-4-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealing-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001533375%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:29:58 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:30:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint-gallons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:02 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97358A105&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297358A105%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-packs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12709 pages (at 53 pages/min), scraped 367 items (at 3 items/min) 2025-11-02 02:30:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/diameter~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-supplies/mounting-location~rack/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gasket-material/material~silicone-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-grade-coatings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rivets/diameter~1-4/ 2025-11-02 02:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-grade-casting-compounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/color~red&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/for-use-with~bases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-supplies/output-voltage~115v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-packs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992574902667&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gasket%20material&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992574902667%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-supplies/output-voltage~5v-dc-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-supplies/mounting-location~rack&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522000%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-supplies/external-power-supply-efficiency-standard~coc-tier-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/for-use-with~chlorine/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5598907428381&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5598907428381%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002380865%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~bases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-supplies/output-voltage~115v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992574902667&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gasket%20material&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992574902667%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-supplies/output-voltage~5v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7686K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227686K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179297%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5598907428381&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5598907428381%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001606727%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/power-supplies/output-voltage~5v-dc/ 2025-11-02 02:30:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=milled-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-supplies/specifications-met~eu-coc-tier-2-efficiency&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:30:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~chlorine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panel-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6718T51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226718T51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992574902667&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gasket%20material&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992574902667%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:31:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/power-supplies/specifications-met~eu-coc-tier-2-efficiency/ 2025-11-02 02:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12760 pages (at 51 pages/min), scraped 369 items (at 2 items/min) 2025-11-02 02:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965544%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/for-use-with~gasoline/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=milling-machine-collet-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/military-specification~ms20426ad5-6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/for-use-with~r-22-refrigerant/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panel-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:23 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/military-specification~ms20470ad8-7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-industry-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~gasoline&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ppsu/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/for-use-with~sodium-bicarbonate-baking-soda/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/military-specifications-met~ms20426ad5-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~refrigerant-r-22&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/military-specifications-met~ms20470ad8-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealtight-conduit/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rivets/military-specifications-met~ms20426ad5-6/ 2025-11-02 02:31:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/seals/for-use-with~refrigerant-r-22/ 2025-11-02 02:31:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rivets/military-specifications-met~ms20470ad8-7/ 2025-11-02 02:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miniature-electric-actuators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ppsu&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~sodium-bicarbonate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seamless-pipe/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/precision-spacers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealtight-conduit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/seals/for-use-with~sodium-bicarbonate/ 2025-11-02 02:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miniature-levels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12805 pages (at 45 pages/min), scraped 370 items (at 1 items/min) 2025-11-02 02:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parts-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parts-dryers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:32:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:16 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/press-fit-bushings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seamless-pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miniature-relief-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parts-trays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressboard/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gear-and-bearing-pullers/thread-size~5-8-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivnuts/thread-size~4-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-fit-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pegboard-baskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rj-25-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2699428871106&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gear%20and%20bearing%20pullers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2699428871106%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334459%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9987127587466&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivnuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9987127587466%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000122836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressboard&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rj45-bulkheads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=penetrating-lubricants/specifications-met~environmentally-acceptable-lubricant-eal-per-vgp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:32:54 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gear-and-bearing-pullers/thread-size~5-8-18&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:32:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/for-use-with~dairy/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/self-aligning-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3756851797609&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=penetrating%20lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3756851797609%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002077972%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/force-gauges/measurement-unit~kilonewtons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/force-gauges/measurement-unit~decanewtons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12852 pages (at 47 pages/min), scraped 370 items (at 0 items/min) 2025-11-02 02:33:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass-filled-ptfe-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1475K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221475K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mitutoyo-103-218-micrometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/for-use-with~dairy&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=self-aligning-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=forcing-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=force-gauges/scale~kilonewtons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=force-gauges/scale~decanewtons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7534361604797&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7534361604797%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964932%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/maximum-pressure~4000-psi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/force-gauges/scale~kilonewtons/ 2025-11-02 02:33:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:33:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/force-gauges/scale~decanewtons/ 2025-11-02 02:33:34 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 500 Internal Server Error 2025-11-02 02:33:34 [scrapy.spidermiddlewares.httperror] (PID: 148) INFO: Ignoring response <500 https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=&srchtxt=&mixedworld=force&workstationdata=true>: HTTP status code is not handled or not allowed 2025-11-02 02:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3844K91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223844K91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/self-tapping-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fork-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/maximum-pressure~750-psi-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=forklift-chain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/maximum-pressure-psi~4000-psi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mixing-nozzles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=forklift-truck-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=go-no-go-gauges/material~tool-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=self-tapping-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/maximum-pressure-psi~750-psi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pressure-gauges/maximum-pressure-psi~4000-psi/ 2025-11-02 02:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4350123131503&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=go%20no%20go%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4350123131503%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510815%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-indicating-film/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mixing-tubes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12904 pages (at 52 pages/min), scraped 372 items (at 2 items/min) 2025-11-02 02:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7534361604797&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7534361604797%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001701889%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobil-velocite-no-3-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fountain-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-indicating-film&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/set-screws/thread-size~1-4-28-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=four-leg-web-slings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-a572-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-b6-stainless-steel-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-teach-pendant-cords/manufacturer-equivalent-model-number~152029-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/set-screws/thread-size~m10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=set-screws/thread-size~1-4-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3733552981249&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robot%20teach%20pendant%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3733552981249%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005188466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-g-11-garolite-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-g-11-laminate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/set-screws/thread-size~1-4-28/ 2025-11-02 02:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=modeling-clay&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3822N12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223822N12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mold-and-mildew-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=set-screws/thread-size~m10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-le-laminate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 12955 pages (at 51 pages/min), scraped 373 items (at 1 items/min) 2025-11-02 02:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-tool-adapters-for-robot-machine-vision-camera-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing/shape~i-beam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4940968732154&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4940968732154%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038594%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971113029698&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=framing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971113029698%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002309136%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=molds/material~babbitt-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=moleskin-bandages&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~compensation-device/for-robot-arm-manufacturer~precise-automation&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=momentary-access-safety-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/process-chillers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=momentary-foot-pedal-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/procunier-1-e-tap-collets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9988569181719&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=molds&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9988569181719%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511236%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=momentary-selector-pilot-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9985525866263&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robotics&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9985525866263%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123607%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002239479%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shaft-aligners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/handles/square-drive-size~1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=process-chillers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=procunier-1-e-tap-collets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grinders/manufacturer-model-number~4000-2-30-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shaft-shims/shaft-type~d-profile/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-aligners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:35:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~tool/manufacturer~abb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=freeze-sprays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=1%22-square-drive-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/abrasive-material~ceramic-alumina&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13005 pages (at 50 pages/min), scraped 373 items (at 0 items/min) 2025-11-02 02:36:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinders/manufacturer~dremel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinders/manufacturer-model-number~4000-2-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9985525866263&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robotics&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9985525866263%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108937%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002320929%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=1%22-square-drive-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001532913%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-shims/shape~d-profile&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7765984901538&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7765984901538%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002263710%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~vacuum-lifter/manufacturer~mitsubishi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grinders/manufacturer-model-number~4000-2-30/ 2025-11-02 02:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6968N111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226968N111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/robotics/robot-cell-component~vacuum-lifter/for-robot-arm-manufacturer~precise-automation/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=1%22-square-drive-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9968617833451&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9968617833451%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310549%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:25 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinders/manufacturer~dremel&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9985525866263&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robotics&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9985525866263%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002096768%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~vacuum-lifter/manufacturer~omron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-011/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 500 Internal Server Error 2025-11-02 02:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:31 [scrapy.spidermiddlewares.httperror] (PID: 148) INFO: Ignoring response <500 https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=1%22-square-drive-handles&srchtxt=&mixedworld=force&workstationdata=true>: HTTP status code is not handled or not allowed 2025-11-02 02:36:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/robotics/robot-cell-component~workstation/for-robot-arm-manufacturer~doosan/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:36:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-035/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9985525866263&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robotics&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9985525866263%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002096768%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002257776%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~vacuum-lifter/manufacturer~precise-automation&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9968617833451&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9968617833451%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310549%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-011inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=frl-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-alignment-couplers/thread-size~3-4-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~workstation/manufacturer~doosan&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-035inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:36:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-011inches/ 2025-11-02 02:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-alignment-couplers/thread-size~3-8-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3106802242081&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rod%20alignment%20couplers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3106802242081%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9985525866263&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robotics&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9985525866263%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116206%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-035inches/ 2025-11-02 02:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-0782/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13052 pages (at 47 pages/min), scraped 374 items (at 1 items/min) 2025-11-02 02:37:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9968617833451&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9968617833451%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310549%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3106802242081&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rod%20alignment%20couplers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3106802242081%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/material~plastic-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=motor-slides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1101K37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221101K37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5206N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225206N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1732707636827&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grommets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1732707636827%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/motors/motor-enclosure-type~totally-enclosed-fan-cooled-tefc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-0782inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3106802242081&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rod%20alignment%20couplers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3106802242081%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ground-handling-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=guarded-shackle-padlocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-0782inches/ 2025-11-02 02:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-111/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5206N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225206N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/motors/mounting-location~flange/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=motors/motor-enclosure-type~totally-enclosed-fan-cooled-tefc-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=guards/material~fiberglass-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3106802242081&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rod%20alignment%20couplers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3106802242081%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-144/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=guards/material~garolite-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-153/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/motors/motor-enclosure-type~totally-enclosed-fan-cooled-tefc-/ 2025-11-02 02:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980676003823&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=guards&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980676003823%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003423545%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-111inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5206N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225206N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980676003823&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=guards&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980676003823%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003432806%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=motors/mounting-location~flange-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-111inches/ 2025-11-02 02:38:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13101 pages (at 49 pages/min), scraped 375 items (at 1 items/min) 2025-11-02 02:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-144inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6780N111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226780N111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-153inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/motors/mounting-location~flange-1/ 2025-11-02 02:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-207/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mounting-paper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:16 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/guides/material~fabric-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-144inches/ 2025-11-02 02:38:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:38:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-153inches/ 2025-11-02 02:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~5-16/diameter~0-3125/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-0145&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-5313/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-207inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-5469/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-124656-helical-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=guides/material~fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548160%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=2l310-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hard-chrome-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-207inches/ 2025-11-02 02:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-3125inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/guides/material~fabric/ 2025-11-02 02:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A297&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A297%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-5313inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-3125inches/ 2025-11-02 02:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~15-16/diameter~0-9375/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-5469inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-15795-826-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~1-0625/diameter~1-1-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-5313inches/ 2025-11-02 02:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuses/current~10-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-5469inches/ 2025-11-02 02:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13148 pages (at 47 pages/min), scraped 376 items (at 1 items/min) 2025-11-02 02:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16995-28-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A297&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A297%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8931638457760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fuses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8931638457760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001514240%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-9375inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16995-4-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~1-0625inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-152&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-9375inches/ 2025-11-02 02:39:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=312a%252f312ae-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-185&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~1-0625inches/ 2025-11-02 02:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fuses/current~325-amp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A297&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A297%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~11-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522752%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001549311%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-228&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~15-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/drive-size~100-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A236&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A236%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A224&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A224%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuses/current~325a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001534640%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2653472997099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20drivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2653472997099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489960%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~11millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:39:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/fuses/current~325a/ 2025-11-02 02:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-3906&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16997-84-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=54155A37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2254155A37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~11millimeters/ 2025-11-02 02:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13198 pages (at 50 pages/min), scraped 378 items (at 2 items/min) 2025-11-02 02:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/drive-size~15-mm-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~2-65-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001533886%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/drive-size~15-mm-1/ 2025-11-02 02:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/drive-size~3-3-4-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~0-031/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A137&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A137%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~1-55-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~10-6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~2-65millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/3d-printer-filaments/material~abs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2653472997099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20drivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2653472997099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003490024%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-4844&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~37-4millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~2-65millimeters/ 2025-11-02 02:40:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~2inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~0-031&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001523045%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~1-55mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~37-4millimeters/ 2025-11-02 02:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~10-6mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~2inches/ 2025-11-02 02:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~15-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/material~abs-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~0-031/ 2025-11-02 02:40:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~1-55mm/ 2025-11-02 02:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A144&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A144%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~10-6mm/ 2025-11-02 02:40:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:40:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/3d-printer-filaments/material~abs-plastic/ 2025-11-02 02:41:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~21-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/material~pett&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/material~plastic-blend&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13245 pages (at 47 pages/min), scraped 382 items (at 4 items/min) 2025-11-02 02:41:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~15-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24693-c275-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8230254771998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20filaments&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8230254771998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548106%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~13-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24693-c300-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8230254771998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20filaments&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8230254771998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511607%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~15-mm/ 2025-11-02 02:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~21-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3462N29&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223462N29%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542673%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-nozzles/body-material~brass-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/fabrication~cold-rolled/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3483N3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223483N3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~21-mm/ 2025-11-02 02:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522979%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971536404295&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971536404295%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001573851%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~42-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~5-mm-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8920K678&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228920K678%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~19-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3462N29&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223462N29%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3695N301&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223695N301%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/fabrication~cold-rolled&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3483N3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223483N3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001553510%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002275085%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=galvanized-steel-register-boots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~42-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:41:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8920K678&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228920K678%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8888K17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228888K17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms27183-51/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3462N29&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223462N29%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~42-mm/ 2025-11-02 02:42:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:42:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~5-mm/ 2025-11-02 02:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13295 pages (at 50 pages/min), scraped 383 items (at 1 items/min) 2025-11-02 02:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3483N3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223483N3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~3l130/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screws/military-specification~ms35207-265/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~no-9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8920K678&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228920K678%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-27183-51-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:23 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gangways&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35275-225-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3l130-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35207-265-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~no-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~t6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:39 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:42:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-nuts/military-specification~ms35691-11/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35276-263-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~no-9/ 2025-11-02 02:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-nuts/military-specification~ms35691-19/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~t6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-nuts/military-specifications-met~ms35691-11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~3-9-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hexagonal-collets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3m-4941-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~t6/ 2025-11-02 02:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13343 pages (at 48 pages/min), scraped 384 items (at 1 items/min) 2025-11-02 02:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-pressure-sodium-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-nuts/military-specifications-met~ms35691-11/ 2025-11-02 02:43:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001527374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-nuts/specifications-met~ms35691-19&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~37-4-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/thread-size~m30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~38-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A276&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A276%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-nuts/specifications-met~ms35691-19/ 2025-11-02 02:43:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001526426%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~4-45-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~4-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536565%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sharps-containers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-strength-adhesives/manufacturer~3m-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51021-36-set-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001543708%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/thread-size~m30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=94353A325&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2294353A325%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shear-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~4-45mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334725%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8920K511&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228920K511%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3m-high-strength-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sharps-containers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~4-45mm/ 2025-11-02 02:43:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~6-25-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:43:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~5-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3m-scotch-brite-abrasive-pads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shear-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shears/for-use-on~polyethylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13391 pages (at 48 pages/min), scraped 387 items (at 3 items/min) 2025-11-02 02:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001531232%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~6-25mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-powered-saws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A322&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A322%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~1-2-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sheets/thickness~50-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-8-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~6-25mm/ 2025-11-02 02:44:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shears/for-use-on~polyethylene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000019175%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5847281553902&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5847281553902%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930094%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~8mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4093A41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224093A41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tig-torch-caps/tig-industry-number~41v24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sheets/thickness~50-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51959-14-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~17-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~14-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~19-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~8mm/ 2025-11-02 02:44:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8336A28&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228336A28%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001461077%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000096913%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142861%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:44:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000032529%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=41v24-tig-torch-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shelving-posts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8189K18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228189K18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4298A23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224298A23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4298A19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224298A19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13440 pages (at 49 pages/min), scraped 390 items (at 3 items/min) 2025-11-02 02:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4298A25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224298A25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~4-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/material~plastic-1/performance~metal-detectable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001528726%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~50-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shelving-posts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gasket-material/military-specification~mil-g-12803/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shields/material~carbon-fiber-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000029865%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51963-65-set-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4066A57&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224066A57%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51967-15-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/thread-size~1-1-4-8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gasket-material/thickness~0-06&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4556A19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224556A19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gasket-material/military-specifications-met~mil-g-12803&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992574902667&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gasket%20material&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992574902667%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001462486%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shields/material~carbon-fiber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gasket-material/military-specifications-met~mil-g-12803/ 2025-11-02 02:45:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shields/material~carbon-fiber/ 2025-11-02 02:45:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51971-7-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:45:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~1-1-8-7-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~4l500/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/thread-size~1-4-28-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/for-use-on~composite-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13491 pages (at 51 pages/min), scraped 396 items (at 6 items/min) 2025-11-02 02:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms3367-6-cable-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334544%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=msds-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003433251%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~4l640/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~1-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:19 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=4l500-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:21 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~1-1-8-7-2&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~1-4-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1%2F4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221%2F4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shims/material~copper-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shims/material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454032%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/thread-size~1-4-28/ 2025-11-02 02:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=4l640-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A744&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A744%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/signs/message~emergency-assembly-point/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~6-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=multiton-pallet-truck-replacement-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1%2F4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221%2F4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/material~copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/material~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000157155%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-032/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/for-use-with~liquid-propane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:46:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shims/material~copper/ 2025-11-02 02:46:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shims/material~plastic/ 2025-11-02 02:46:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=muster-point-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-03/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:47:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1%2F4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221%2F4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964742%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~5l490/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shims/thickness~0-006/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nail-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13540 pages (at 49 pages/min), scraped 397 items (at 1 items/min) 2025-11-02 02:47:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-032inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=77495A21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2277495A21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-115/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-03inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:21 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1149-c0463r-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-032inches/ 2025-11-02 02:47:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5l490-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/thickness~0-006&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-03inches/ 2025-11-02 02:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~5l910/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~19-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas1149-f0863p-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001464488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-115inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shims/thickness~0-226/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~5vx740/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1149-fn616p-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-115inches/ 2025-11-02 02:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shims/thickness~2-9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5l910-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-81&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1149-f0863p-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-296875inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/thickness~0-226&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507514%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5vx740-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-296875inches/ 2025-11-02 02:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13585 pages (at 45 pages/min), scraped 398 items (at 1 items/min) 2025-11-02 02:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/thickness~2-9-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roller-chain-links/material~steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001467408%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/for-use-with~electrical-terminals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89965K387&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289965K387%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:48:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:48:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=needle-point-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001467501%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1683557458441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=roller%20chain%20links&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1683557458441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roller-chain-spring-clips/roller-chain-trade-number~25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shock-absorbing-mats/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=needles/needle-gauge~21&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3412378825389&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=roller%20chain%20spring%20clips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3412378825389%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000010068%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/washers-for-electrical-terminals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001467408%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2577530562040&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=needles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2577530562040%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148770%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%228972104892258%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~3-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-6-30-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-6-20-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=90-degree-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-diameter~20-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=93720A011&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2293720A011%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shock-absorbing-mats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498994%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001467408%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2577530562040&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=needles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2577530562040%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148770%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50415K59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250415K59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13637 pages (at 52 pages/min), scraped 399 items (at 1 items/min) 2025-11-02 02:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498330%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-diameter~20mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=93720A011&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2293720A011%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:14 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-l17-30-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2577530562040&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=needles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2577530562040%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148770%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a-plate-sprockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rollers/roller-diameter~20mm/ 2025-11-02 02:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-diameter~72-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~6inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=93720A011&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2293720A011%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~7-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/receptacles/nema-type~l7-30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a286-super-alloy-hex-flange-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:35 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:49:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shoulder-bolts/thread-size~m24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-diameter~90-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~6inches/ 2025-11-02 02:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505314%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-diameter~72mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000114647%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:49:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-l7-30-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shoulder-bolts/thread-size~m24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rollers/roller-diameter~72mm/ 2025-11-02 02:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~11-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1470T661&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221470T661%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=neoprene%252fepdm%252fsbr-foam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-011/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7944149739131&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7944149739131%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334859%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13688 pages (at 51 pages/min), scraped 400 items (at 1 items/min) 2025-11-02 02:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~17-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/thread-type~npt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrasive-blasting-hoods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000168645%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-015/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-011&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nickel-alloy-718-rounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~20-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2297T32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222297T32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-011/ 2025-11-02 02:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nickel-chromium-wire-cloth&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000139833%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sight-glass-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-0150&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrasive-mounted-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-029/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-037/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12975N12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212975N12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-0150/ 2025-11-02 02:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:50:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sight-glass-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-043/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrasive-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-029&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13735 pages (at 47 pages/min), scraped 403 items (at 3 items/min) 2025-11-02 02:51:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-037&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-1003/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abs%252fpvc-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-029/ 2025-11-02 02:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~cyclopropane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-037/ 2025-11-02 02:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~30-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-0430&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~f2b-012-rm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-114/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-123/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-1003&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~hydrogen-bromide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-0430/ 2025-11-02 02:51:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123930%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-1003/ 2025-11-02 02:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967470%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/for-use-with~sodium-carbonate-soda-ash/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~30mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-114&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/silicon-nitride-ceramic/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-1230&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=79215A73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2279215A73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rollers/roller-width~30mm/ 2025-11-02 02:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-114/ 2025-11-02 02:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:51:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-1230/ 2025-11-02 02:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~sodium-carbonate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~fyt-2-rm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13783 pages (at 48 pages/min), scraped 404 items (at 1 items/min) 2025-11-02 02:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~47-7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silicon-nitride-ceramic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/silicone-paste/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nipples/for-use-with~sodium-carbonate/ 2025-11-02 02:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/thread-size~0-855-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116350%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~8-19-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000025662%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/silicone-rubber-bumpers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=access-latch-fasteners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455475%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:24 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000141222%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2281T45&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222281T45%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silicone-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=79215A413&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2279215A413%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~5-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rolling-stools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silicone-rubber-bumpers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~npl-22&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~5-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002131197%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/single-acting-cylinders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alligator-belt-lacing/lacing-number~45us/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=romex-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~p2bl-108-wf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~5-5-mm/ 2025-11-02 02:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dry-film-thickness-gauges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13829 pages (at 46 pages/min), scraped 407 items (at 3 items/min) 2025-11-02 02:53:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-quick-disconnect-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002103605%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-homopolymer-rod-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~5-mm/ 2025-11-02 02:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=single-acting-cylinders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=no-45us-alligator-belt-lacing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/coating-thickness-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=no-hub-pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rope/diameter~0-031&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%228550323773032%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rope/diameter~0-047&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~sf-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sintered-bronze/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7090012887875&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rope&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7090012887875%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001545401%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2586N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222586N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acid-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~sfc-208&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7090012887875&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rope&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7090012887875%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001528292%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108825%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acme-mounting-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002076324%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-cutters/performance~nonmagnetic-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sintered-bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slatwall-bins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13879 pages (at 50 pages/min), scraped 408 items (at 1 items/min) 2025-11-02 02:54:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slatwall-brackets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slatwall-panels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acme-threaded-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rope/material~eptfe-fiber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~st-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acrylic-anchors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonmagnetic-wire-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slatwall-bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002097103%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slatwall-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slatwall-panels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~sth-27-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rope/material~eptfe-fiber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rope-seals/for-use-with~solvents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002111671%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drinking-fountain-replacement-parts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~ucp203&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9785102052963&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rope%20seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9785102052963%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179285%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/measures~line-width&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rosebud-heads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-level-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:54:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulated-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003916893%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002096235%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 02:55:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 02:55:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/drinking-fountain-replacement-parts-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13926 pages (at 47 pages/min), scraped 408 items (at 0 items/min) 2025-11-02 02:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1755T17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221755T17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/measurement-unit~bar/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nozzles/drinking-fountain-replacement-parts-4/ 2025-11-02 02:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation/material~nickel-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/spray-application~high-pressure&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rotary-tool-chucks/manufacturer~dremel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/spray-application~no-drip&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/scale~bar&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation/performance~weather-resistant-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/material~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001974559%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/material~nickel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001973666%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauges/scale~bar/ 2025-11-02 02:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/insulation/material~brass/ 2025-11-02 02:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3234K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223234K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nsf%252fansi-standard-solenoid-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tool-chucks/for-manufacturer~dremel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/insulation/material~nickel/ 2025-11-02 02:55:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4846T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224846T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slotted-sheet-metal-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iron-metric-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/performance~weatherproof-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6504433171172&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20chucks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6504433171172%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002147818%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:55:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insulation&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116389%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rotary-tool-safety-guards/manufacturer-model-number~100-n-7-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iron-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tool-grips/manufacturer-model-number~3000&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 13973 pages (at 47 pages/min), scraped 411 items (at 3 items/min) 2025-11-02 02:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/small-parts-picks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/smc-compressed-air-regulators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slotted-sheet-metal-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tool-router-bits/manufacturer-model-number~8250&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5600772794809&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20grips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5600772794809%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004703630%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6504433171172&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20chucks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6504433171172%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002147818%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3176430263648&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20router%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3176430263648%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004703627%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tool-safety-guards/manufacturer-model-number~100-n-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=small-parts-picks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=smc-compressed-air-regulators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rotary-tool-safety-guards/manufacturer-model-number~100-n-7/ 2025-11-02 02:56:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tool-wheel-brushes/manufacturer-model-number~3965-02&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/hex-nut-profile~narrow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6504433171172&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20chucks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6504433171172%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002147818%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-unions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/thread-size~m12-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~9-16-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6174913493338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20wheel%20brushes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6174913493338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321784%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000105984%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334596%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotating-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:52 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/hex-nut-profile~narrow&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:56:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~2-1-4-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~m12-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~3-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/so-cable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-type~unjc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454471%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14025 pages (at 52 pages/min), scraped 411 items (at 0 items/min) 2025-11-02 02:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000009660048%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451720%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~7-8-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-belts/diameter~7-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002394459%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~2-1-4-12&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gears/pitch~9-525-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=general-purpose-bronze-acme-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:20 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~3-12&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=so-cable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6178916986877&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=round%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6178916986877%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550872%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=93904A100&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2293904A100%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000009660048%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4087963384680&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gears&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4087963384680%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000117418%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:30 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~7-8-14&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:57:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:57:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lags/for-use-in~concrete&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lags/thread-size~m12-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9974560404121&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lags&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9974560404121%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930049%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000009660048%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laminate-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-1-2-4-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~1-12-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lags/thread-size~m12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=additive-manufacturing-machines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14076 pages (at 51 pages/min), scraped 412 items (at 1 items/min) 2025-11-02 02:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/lags/thread-size~m12/ 2025-11-02 02:58:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~1-312-18-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-accelerators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A771&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A771%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-accessories&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~2-1-4-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-round-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nuts/thread-size~1-12/ 2025-11-02 02:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=routing-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-312-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rtd-heads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A771&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A771%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454471%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nuts/thread-size~1-312-18/ 2025-11-02 02:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~3-8-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:47 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~2-1-4-12&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:58:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass-cleaning-wipes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/solder-end-gate-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rtds/environment~high-pressure-washdown&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber/color~light-gray-white/pattern~marble/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451197%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber/material~rubber-1/material~buna-n-graphite/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A771&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A771%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-screws/thread-type~metric-rounded-trapezoidal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m155&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7410228447115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rtds&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7410228447115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000039253%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6350K702&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226350K702%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14126 pages (at 50 pages/min), scraped 412 items (at 0 items/min) 2025-11-02 02:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1206504422411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lead%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1206504422411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002393713%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solder-end-gate-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451540%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3866K62&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223866K62%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/color~marbled-light-gray&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:17 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 02:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97783A150&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297783A150%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m42&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rubber/color~marbled-light-gray/ 2025-11-02 02:59:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 02:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6350K702&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226350K702%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/material~rubber/material~graphite-buna-n-rubber-blend&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453388%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber/mount-type~tack/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453496%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rubber/material~rubber/material~graphite-buna-n-rubber-blend/ 2025-11-02 02:59:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/material~rubber-1/material~synthetic-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:42 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m42&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:59:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-base-type~wedge&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~rubber-2/for-use-on~epdm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6350K702&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226350K702%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:47 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m48&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 02:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~pg-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000164949%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511614%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/mount-type~hammer-in&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001931112%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930459%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002940493%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~85/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/shape~doughnut&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454988%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 02:59:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rubber/mount-type~hammer-in/ 2025-11-02 03:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6932K53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226932K53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-type~npsm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:00:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002311290%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14178 pages (at 52 pages/min), scraped 414 items (at 2 items/min) 2025-11-02 03:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/solenoids/body-material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3185K93&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223185K93%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000151504%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glides/mount-type~rest-on/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~85&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/shape~doughnut&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/solenoids/for-use-with~isopropyl-alcohol/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glazing-point-driver-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brooms/bristle-material~rubber-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~85/ 2025-11-02 03:00:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-bands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solenoids/body-material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/solenoids/for-use-with~liquid-argon/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locks/material~bronze-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glides/mount-type~freestanding&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/solenoids/body-material~steel/ 2025-11-02 03:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solenoids/for-use-with~isopropyl-alcohol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-brooms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-quick-turn-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gloves/color~gold-black/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9952801634820&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glides&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9952801634820%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010917%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solenoids/for-use-with~liquid-argon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=168950821422&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B168950821422%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965701%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screw-mount-rubber-feet/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-ring-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locks/material~bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=168950821422&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B168950821422%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968165%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4744K24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224744K24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/locks/material~bronze/ 2025-11-02 03:01:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/color~black-gold&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50585K23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250585K23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14224 pages (at 46 pages/min), scraped 416 items (at 2 items/min) 2025-11-02 03:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-feet/mount-type~screw&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/color~blue-white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/color~fluorescent-green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gloves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002193354%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/color~off-white-fluorescent-orange&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8031429930803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber%20feet&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8031429930803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003011043%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gloves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002191248%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gloves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002196894%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~102&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7849N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227849N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-4211&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gloves/color~off-white-fluorescent-orange/ 2025-11-02 03:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-raw-materials&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=57945T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2257945T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000082086%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-rolls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gloves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002191248%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K316&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K316%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/thickness~0-004&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-arm-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-clearance-padlocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-598&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~256&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gloves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002191248%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gloves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001463522%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-handles/material~nylon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-7452&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000052185%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14280 pages (at 56 pages/min), scraped 421 items (at 5 items/min) 2025-11-02 03:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=52555T643&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2252555T643%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8767262773158&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adjustable%20handles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8767262773158%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511113%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~274&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:16 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/for-use-on~asphalt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/for-housing-id~1-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000096461%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~275&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-twist-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/soap-cartridges/manufacturer~rubbermaid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/for-housing-id~2-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K373&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K373%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000112308%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-e-90fl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-housing-id~1-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:42 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-housing-id~2-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubbermaid-soap-cartridges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000159382%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/for-housing-id~40-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000112308%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:02:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:02:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000051913%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~105-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-threadlockers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/screw-size~m180/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14328 pages (at 48 pages/min), scraped 423 items (at 2 items/min) 2025-11-02 03:03:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=low-headroom-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001622853%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/food-grade-penetrating-lubricants/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-housing-id~40mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000112308%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:18 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~105-ft&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~22-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/spacers/for-housing-id~40mm/ 2025-11-02 03:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-screw-size~m180&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/formulation~polyester-plastic/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-use-on~plastic-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001628457%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/environment-1~food-and-beverage/lubricant-type~penetrating-lubricant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjusting-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:34 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:03:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/spacers/for-screw-size~m180/ 2025-11-02 03:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930351%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~dupont-non-stick-dry-film-lubricant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/lubricants/environment-1~food-and-beverage/lubricant-type~penetrating-lubricant/ 2025-11-02 03:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1923A26&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221923A26%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~33-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/formulation~polyester&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:46 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-use-on~plastic-1&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:03:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002142752%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001584821%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aerosol-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/glue/formulation~polyester/ 2025-11-02 03:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~45-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=made-to-order-shaft-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:00 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~33-ft&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-actuators/actuation-mechanism~air-rotate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001619085%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~15820&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14384 pages (at 56 pages/min), scraped 424 items (at 1 items/min) 2025-11-02 03:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~201&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731925765182&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20actuators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731925765182%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002055663%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:13 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~45-ft&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002142475%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/material~stainless-steel-2/material~321-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002074107%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6508K101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226508K101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnets/material~plastic-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oak-workbench-tops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74575A33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274575A33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511875%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~27010&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6937T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226937T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3815738403502&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=magnets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3815738403502%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002082573%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~290&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6508K101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226508K101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=odor-resistant-blue-polyethylene-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74575A33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274575A33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/thread-size~1-967-18-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511875%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002149985%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3815738403502&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=magnets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3815738403502%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:04:58 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6508K101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226508K101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/thread-size~3-4-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74575A33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274575A33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/thread-size~1-967-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511875%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14435 pages (at 51 pages/min), scraped 425 items (at 1 items/min) 2025-11-02 03:05:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~330&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/thread-size~3-8-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3815738403502&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=magnets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3815738403502%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulon-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/spacers/thread-size~1-967-18/ 2025-11-02 03:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002096815%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/thread-size~3-4-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:23 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:05:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:05:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1747A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221747A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/washer-type~angled-edge/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/thread-size~3-8-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=masks/for-use-with~acid-gas&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454348%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rwma-class-2-copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3028195520048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=masks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3028195520048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179315%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-cylinder-clevis-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-air-compressor-regulators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/washer-type~angled-edge&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-bevel-gears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5541T294&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225541T294%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~4541&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001982091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/manufacturer~starrett-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14485 pages (at 50 pages/min), scraped 427 items (at 2 items/min) 2025-11-02 03:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002091728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95990A101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295990A101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saddle-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~4sc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1818A49&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221818A49%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~5091&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/manufacturer~starrett&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-cylinders/bore-size~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/glue/manufacturer-model-number~4sc/ 2025-11-02 03:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/specialty-metric-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002095816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/oil/manufacturer~starrett/ 2025-11-02 03:06:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4216247920660&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4216247920660%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000058430%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sae-80-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1832A22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221832A22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~krytox-gpl-105&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-engraving-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~609&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=specialty-metric-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~638&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114825%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002086961%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002105060%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-cylinders/stroke-length~1-5/stroke-length~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-cages&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14540 pages (at 55 pages/min), scraped 430 items (at 3 items/min) 2025-11-02 03:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-hose-to-pipe-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-cylinders/stroke-length~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114825%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~7090&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4216247920660&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4216247920660%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000151608%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000079477%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-jobsite-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4216247920660&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4216247920660%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000124505%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~mobilgear-600-xp-68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079278%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~7649&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002146215%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114825%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099590%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-pans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~80628&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-power-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-filter-frames&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-push-on-hose-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002091302%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002146215%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~molykote-d-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:57 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~80628&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:07:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/oil/trade-name~molykote-d-paste/ 2025-11-02 03:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~molykote-dx-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-filters/hvac-filter-trade-size~12-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14597 pages (at 57 pages/min), scraped 430 items (at 0 items/min) 2025-11-02 03:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-scales&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002146215%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002105331%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~synxtreme-fg-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=851279397977&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B851279397977%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002057662%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~ca5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~tapmatic-natural&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-filters/hvac-filter-trade-size~14-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136461%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~wd-40-easy-reach&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099572%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002090905%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=851279397977&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B851279397977%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002049210%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-glasses/welding-lens-type~standard&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2242N21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222242N21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002088090%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=75445A73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2275445A73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tie-down-anchors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=751624917026&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=safety%20glasses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B751624917026%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001982373%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-submersible-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1347K51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221347K51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-filters/hvac-filter-trade-size~21-21&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spline-l-keys/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=851279397977&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B851279397977%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005043911%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/fitting-connection~without-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metals/shape~slot&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-skimmer-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~p591&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-tie-down-anchors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spline-l-keys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14649 pages (at 52 pages/min), scraped 433 items (at 3 items/min) 2025-11-02 03:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-006&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1062337176277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=metals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1062337176277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310471%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002094781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oilers/for-use-with~hydraulic-fluid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/connection-style~without-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oilers/for-use-with~citric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496502%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-crosses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7442A21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227442A21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/air-hose/connection-style~without-fittings/ 2025-11-02 03:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4227901971716&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oilers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4227901971716%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967437%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K612&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K612%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sponge-mats/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oilers/for-use-with~hydraulic-fluid-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~rtv103&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saw-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/oilers/for-use-with~hydraulic-fluid-1/ 2025-11-02 03:09:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121325%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~si-5049&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oilers/for-use-with~sodium-hypochlorite-bleach&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue-bottle-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sponge-mats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7545A461&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227545A461%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002076626%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-gun-nozzles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=salinity-monitors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4227901971716&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oilers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4227901971716%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984462%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scoops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74885A12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274885A12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=open-wedge-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14704 pages (at 55 pages/min), scraped 436 items (at 3 items/min) 2025-11-02 03:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sanitizer-cartridges/manufacturer~gojo/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spray-bottles/tank-material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mica-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screen-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=optically-clear-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-319&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gojo-sanitizer-cartridges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=optical-wipes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-345&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-bottles/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gore-joint-sealant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screw-bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496691%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497736%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/spray-bottles/material~aluminum/ 2025-11-02 03:10:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K736&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K736%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-436&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6100K364&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226100K364%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miter-gears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506791%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oscillating-saws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oscillating-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=out-of-service-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sanding-rolls/abrasive-material~ceramic-alumina/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-902&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14755 pages (at 51 pages/min), scraped 439 items (at 3 items/min) 2025-11-02 03:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89965K491&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289965K491%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001499929%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/air-hose/id~0-5mm/ 2025-11-02 03:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/outlets/current-voltage~1-8-amp-15v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~1-152&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanding-rolls/abrasive-material~aluminum-oxide-ceramic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sandpaper/form~bit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K768&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K768%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screw-tip-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497370%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sanding-rolls/abrasive-material~aluminum-oxide-ceramic/ 2025-11-02 03:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2386623665772&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sandpaper&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2386623665772%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002127123%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/current~150-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~2-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/current~1-8-a-15-v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~t9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978812115307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978812115307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001526641%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497758%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/outlets/current~1-8-a-15-v-dc/ 2025-11-02 03:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K775&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K775%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sandpaper/grit~2400&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2386623665772&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sandpaper&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2386623665772%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002127123%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/current-voltage~0-28-amp-100v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8030K81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228030K81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:11:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K623&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K623%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/mounting-location~surface&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/voltage~250v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2386623665772&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sandpaper&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2386623665772%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003150727%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~t9-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978812115307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978812115307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002006158%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14808 pages (at 53 pages/min), scraped 444 items (at 5 items/min) 2025-11-02 03:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978812115307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978812115307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002236973%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978812115307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978812115307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003552938%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/drive-size~t9-1/ 2025-11-02 03:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4952A218&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224952A218%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2386623665772&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sandpaper&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2386623665772%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002127123%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3741T13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223741T13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/length~17-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7316K15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227316K15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~16-to-120&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ac-motors/environment~sanitary/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sandpaper-sticks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~200-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/drive-size~t27/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004099105%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:33 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:12:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~17-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001622923%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5626K99&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225626K99%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outside-micrometers/manufacturer-model-number~103-922&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/air-hose/length~17-ft-/ 2025-11-02 03:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-ac-motors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35275-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1979T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221979T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/drive-size~t27-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8260587697623&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outside%20micrometers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8260587697623%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002135966%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:12:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outside-micrometers/manufacturer-model-number~223-125&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screws/drive-size~t27-1/ 2025-11-02 03:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5626K99&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225626K99%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14858 pages (at 50 pages/min), scraped 448 items (at 4 items/min) 2025-11-02 03:13:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35649-machine-screw-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8260587697623&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outside%20micrometers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8260587697623%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002151398%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-mounts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/material~fiberglass-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/od~12-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5626K99&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225626K99%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/material~titanium-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outside-micrometers/manufacturer-series~733&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose-chucks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003423545%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000022471%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-inflators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8260587697623&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outside%20micrometers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8260587697623%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005015215%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000097208%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~1-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5265K49&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225265K49%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:42 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334356%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003423545%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~1-8-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-round-head-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~8-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1352-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nas1352n-4-20-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14910 pages (at 52 pages/min), scraped 449 items (at 1 items/min) 2025-11-02 03:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overload-relays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=granite-parallels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334719%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003423545%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000060837%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1352n-4-20-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-620-c10l-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas1351c-3-12-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sash-paint-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-powered-punches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-powered-scalers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~0-38/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saw-blade-stabilizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/for-use-on~stone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930162%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oxyacetylene-welding-torch-accessories&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~1-91/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saw-sharpening-files&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~0-38inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas1352c-3-8-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas1352c-4-8-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealers/for-use-on~plastic-1/for-use-on~abs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/springs/length~0-38inches/ 2025-11-02 03:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 14963 pages (at 53 pages/min), scraped 449 items (at 0 items/min) 2025-11-02 03:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gravel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~530&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nasm15795/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~1-91inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/length~25-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/for-use-on~abs-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/springs/length~1-91inches/ 2025-11-02 03:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002094232%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/for-use-on~plastic-1/for-use-on~abs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/for-use-on~bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cam-latches/handle-style~paddle/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=airlocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nasm-15795-805-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sealers/for-use-on~abs-plastic/ 2025-11-02 03:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984350057510&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984350057510%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930351%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929784%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grease&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929963%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~25-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=packing-slip-envelopes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/air-hose/length~25-ft-/ 2025-11-02 03:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paddle-handle-cam-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alignment-locks/finish~titanium-nitride-tin-coated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/allen-key-wrenches/performance~nonsparking-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=neoprene-foam-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alligator-belt-lacing/lacing-number~15s/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4929613489696&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=alignment%20locks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4929613489696%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001535325%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sbr-rubber-bar-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pads/material~cork-fiber-reinforced-pvc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15009 pages (at 46 pages/min), scraped 449 items (at 0 items/min) 2025-11-02 03:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=allen-screws/material~glass-filled-nylon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=neoprene-vibration-damping-pads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:13 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alignment-locks/finish~titanium-nitride-tin-coated&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=allen-key-wrenches/safety-properties~nonsparking&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7325958487577&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=allen%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7325958487577%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509878%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alligator-belt-lacing/lace-number~15s&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seamless-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3199319696857&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=allen%20key%20wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3199319696857%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002237274%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pads/material~pvc-cork-blend&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/security-boxes/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~lb-8009-heavy-duty-anti-seize&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=91221A110&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2291221A110%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/alligator-belt-lacing/lace-number~15s/ 2025-11-02 03:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=secondary-containments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pads/material~pvc-cork-blend/ 2025-11-02 03:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grease&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nickel-screens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3199319696857&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=allen%20key%20wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3199319696857%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002237274%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=security-boxes/material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/scales/measurement-unit~ra/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=91221A110&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2291221A110%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-levers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pail-and-drum-strainers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/security-boxes/material~steel/ 2025-11-02 03:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminized-foam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3199319696857&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=allen%20key%20wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3199319696857%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002237274%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scales/scale~ra&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=91221A110&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2291221A110%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15062 pages (at 53 pages/min), scraped 449 items (at 0 items/min) 2025-11-02 03:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nitrile-foam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~molykote-g-0050-fm-white-ep&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/scales/scale~ra/ 2025-11-02 03:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scissor-arms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-bars/material~aluminum-2/material~metapor-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grease&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002128205%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-angles/material~5086-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:25 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1445K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221445K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992004840104&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20angles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992004840104%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510747%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-bronze-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scotch-brite-abrasive-pads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-powered-sanders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-bars/material~aluminum/material~metapor-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4039T14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224039T14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint-can-openers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-cases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-extensions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9990216947512&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20bars&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9990216947512%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510078%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/shaft-diameter~7-8-to-15-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease-fitting-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8189K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228189K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4039T14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224039T14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screen-door-stops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15114 pages (at 52 pages/min), scraped 450 items (at 1 items/min) 2025-11-02 03:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-purifiers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/for-shaft-diameter~7-8-to-15-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-mounting-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease-fitting-relocation-blocks/mount-type~magnetic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8189K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228189K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4039T14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224039T14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/for-shaft-diameter~7-8-to-15-16/ 2025-11-02 03:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4483663069189&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grease%20fitting%20relocation%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4483663069189%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003011040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-filled-epoxies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screens/material~fabric-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-6%252f6-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1693K8&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221693K8%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8189K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228189K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:34 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-lock-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease-hose-swivels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/id~17-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease-fitting-assortments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screens/material~fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screens/material~fabric/ 2025-11-02 03:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-pipe-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=painted-aluminum-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-supports&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/id~17-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screens/shape~rectangle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease-fitting-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pallet-bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15168 pages (at 54 pages/min), scraped 451 items (at 1 items/min) 2025-11-02 03:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~0-016&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001500669%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5254672608834&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screens&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5254672608834%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002309280%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pallet-containers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/id~4-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9495772993241&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20plates&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9495772993241%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001455405%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2298K64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222298K64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pallet-levelers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-mesh-abrasives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89015K111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289015K111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screw-conveyor-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-supply&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/id~4-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2298K64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222298K64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pallet-rack-drop-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/thread-size~5-16-24-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:47 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:19:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507239%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334704%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/thread-size~m6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3764N175&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223764N175%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2298K64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222298K64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/keyway-width~5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pallet-truck-replacement-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aircraft-cable-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pan-lids&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000148470%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinders/manufacturer-model-number~100pt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15221 pages (at 53 pages/min), scraped 453 items (at 2 items/min) 2025-11-02 03:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:17 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7765984901538&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7765984901538%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002129775%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/keyway-width~5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sheet-metal-deburring-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease/trade-name~molykote-316-silicone-release-spray/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000148470%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinders/manufacturer-model-number~763625-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/keyway-width~5mm/ 2025-11-02 03:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/material~aluminum-2/material~5083-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-balls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panel-mount-timer-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7765984901538&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7765984901538%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002085407%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/number-of-teeth~48/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/number-of-teeth~44/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510208%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~molykote-316-silicone-release-spray&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000148470%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/panels/material~leather-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4058T71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224058T71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110471%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-tank-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/number-of-teeth~48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/number-of-teeth~44&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-removal-air-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4328T591&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224328T591%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paper%252fphenolic-laminate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15271 pages (at 50 pages/min), scraped 454 items (at 1 items/min) 2025-11-02 03:21:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001527601%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001517006%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panels/material~leather&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/panels/material~leather/ 2025-11-02 03:21:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/pitch~9-525-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/roller-chain-trade-number~16b-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=orifice-valves/specifications-met~asme-b16-36&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~0-111/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~0-118/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdriver-trays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000117418%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shim-stock-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8232313390756&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=orifice%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8232313390756%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005072577%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~0-102&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/roller-chain-trade-size~16b-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5291N12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225291N12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screwdrivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003488430%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outside-micrometers/manufacturer-model-number~148-502&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~0-111-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/double-expansion-anchors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/roller-chain-trade-number~80/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~0-118-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/roller-chain-trade-size~16b-2/ 2025-11-02 03:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8260587697623&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outside%20micrometers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8260587697623%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002125150%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/drive-size~0-111-1/ 2025-11-02 03:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:21:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/drive-size~0-118-1/ 2025-11-02 03:21:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mountable-cable-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5291N12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225291N12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/female-threaded-anchors-for-block-and-brick-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/roller-chain-trade-size~80-ss&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15321 pages (at 50 pages/min), scraped 455 items (at 1 items/min) 2025-11-02 03:22:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screwdrivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489937%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~17-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outside-micrometers/manufacturer-model-number~s909z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/anchors/female-threaded-anchors-for-block-and-brick-8/ 2025-11-02 03:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/roller-chain-trade-size~80-ss/ 2025-11-02 03:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/thread-size~m20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/mountable-cable-ties-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5291N12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225291N12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screwdrivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489948%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8260587697623&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outside%20micrometers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8260587697623%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002109640%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shock-struts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9726243375694&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=anchors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9726243375694%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screwdrivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489937%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/anchors/mountable-cable-ties-6/ 2025-11-02 03:22:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shoe-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:40 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/packing/material~wood-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~ep16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/square-drive-socket-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screwdrivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489937%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/shape-number~w178&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=packing/material~wood&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~ep16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pads/material~fabric-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shoulder-bolts/thread-size~10-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002102931%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-drive-socket-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/shape-number~w191&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15372 pages (at 51 pages/min), scraped 455 items (at 0 items/min) 2025-11-02 03:23:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/packing/material~wood/ 2025-11-02 03:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/square-head-specialty-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/drive-size~ep16-1/ 2025-11-02 03:23:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:23:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951811382783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pads&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951811382783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511101%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7944149739131&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shoulder%20bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7944149739131%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000139805%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:14 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/shape-number~w178&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:23:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004751766%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~ip4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screwdrivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489964%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-head-specialty-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum/material~5005-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=angle-rings-for-exhaust-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pda-styli&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sight-feed-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004751766%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=54305A17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2254305A17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510051%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sight-glass/for-use-with~hydraulic-fluid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/squares/material~carbon/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6062K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226062K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/squares/material~graphene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/panels/material~fabric-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6677440772530&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sight%20glass&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6677440772530%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964825%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004751766%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~0-031/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~0-039/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panels/material~nickel-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=squares/material~carbon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15419 pages (at 47 pages/min), scraped 457 items (at 2 items/min) 2025-11-02 03:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=squares/material~graphene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panels/material~fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paper/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9962166674953&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=panels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9962166674953%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000088769%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958961006685&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958961006685%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509962%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~0-0310&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ansi-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958961006685&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958961006685%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510437%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/panels/material~fabric/ 2025-11-02 03:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~0-0390&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8826K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228826K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~0-071/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~0-0310/ 2025-11-02 03:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paper/material~wood-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7456N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227456N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~0-0390/ 2025-11-02 03:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paper/material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~sr69/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~0-094/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/paper/material~steel/ 2025-11-02 03:24:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8826K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228826K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~0-071&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pem-style-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~0-133/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paper/material~wood&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=single-turn-retaining-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~0-0940&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:24:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~0-071/ 2025-11-02 03:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sr69-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/paper/material~wood/ 2025-11-02 03:25:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stackable-totes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8826K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228826K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15465 pages (at 46 pages/min), scraped 458 items (at 1 items/min) 2025-11-02 03:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=antiseize-lubricant-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~0-0940/ 2025-11-02 03:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grinding-wheels/for-use-on~carbon-fiber-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~0-1330&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~1-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~1-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~0-1330/ 2025-11-02 03:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stackable-totes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:25:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=skimmer-nets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slam-catches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/for-use-on~carbon-fiber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=petg-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grinding-wheels/for-use-on~carbon-fiber/ 2025-11-02 03:25:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~1-1-2/ 2025-11-02 03:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-anchors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/for-use-on~plaster&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~1-16/ 2025-11-02 03:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slatwall&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~5-mm-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929774%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=appliance-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slide-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-anchors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=phillips-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-in-torx-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15515 pages (at 50 pages/min), scraped 458 items (at 0 items/min) 2025-11-02 03:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slide-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-cam-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~ep18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~ip27/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~5-mm/ 2025-11-02 03:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~no-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pfa-duct&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-cam-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~ip27&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=area-lights/light-technology~halogen&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~ep18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pg11-taps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=small-parts-cases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grip-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~ip27/ 2025-11-02 03:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7748475325721&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=area%20lights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7748475325721%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~ep18/ 2025-11-02 03:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~no-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=14685K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2214685K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-089&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~no-8/ 2025-11-02 03:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grip-strut-grating&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grip-struts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001519015%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grippers/gripping-method~angular/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=arrow-compatible-staples&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15563 pages (at 48 pages/min), scraped 459 items (at 1 items/min) 2025-11-02 03:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=articulating-ladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gripper-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/drive-size~0-125&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:14 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-089&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grippers/grip-style~knurled-head/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489875%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/drive-size~0-9-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filter-bags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fulcrum-screws/material~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grippers/grip-style~angular&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=phosphate-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003488402%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=photo-eye-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grippers/grip-style~angular/ 2025-11-02 03:27:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grippers/grip-style~knurled&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:27:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=snap-ring-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=asme-filter-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-fulcrum-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grippers/grip-style~knurled/ 2025-11-02 03:27:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/drive-size~t3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=piano-hinges/material~carbon-fiber-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pick-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=piano%20hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003432842%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grit-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=picks/tip-style~lift&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=astm-b370-copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15614 pages (at 51 pages/min), scraped 459 items (at 0 items/min) 2025-11-02 03:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/drive-size~t3-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-177&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2183N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222183N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5551314863993&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=picks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5551314863993%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243007%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=socket-extensions/square-drive-size~3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/hole-diameter~1-1-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screws/drive-size~t3-1/ 2025-11-02 03:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-193&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001524012%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9943296882328&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=socket%20extensions&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9943296882328%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000056258%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-machine-tables/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cordless-telephone-batteries/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001551451%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:30 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-177&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=socket-screws/thread-direction~right-hand&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/for-hole-diameter~1-1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:39 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-193&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5340812667566&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=socket%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5340812667566%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000138654%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/hole-diameter~15-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=attic-exhaust-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grommets/for-hole-diameter~1-1-16/ 2025-11-02 03:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-machine-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=at%26t-compatible-cordless-phone-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/material~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/hole-diameter~6-2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=auto-transformers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:28:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screws/material~plastic/ 2025-11-02 03:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/for-hole-diameter~15-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-275&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/staplers/office-stapler-type~automatic/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15660 pages (at 46 pages/min), scraped 460 items (at 1 items/min) 2025-11-02 03:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-pipe-unions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grommets/for-hole-diameter~15-64/ 2025-11-02 03:29:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:29:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542152%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/for-hole-diameter~6-2mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=automatic-tape-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=automotive-circuit-breakers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~1-000-mm/diameter~1-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grommets/for-hole-diameter~6-2mm/ 2025-11-02 03:29:23 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-275&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=automatic-staplers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solder/for-station-manufacturer-model-number~wtcpn&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-pipe-unions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001533446%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542841%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~ax22/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pinch-valves/body-material~acetal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~1-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/solder/for-station-manufacturer-model-number~wtcpn/ 2025-11-02 03:29:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/drive-size~t7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:37 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~1-000-mm/diameter~1-mm&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-shelving-brackets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000030183%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~ax71/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=soldering-irons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ax22-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pinch-valves/body-material~acetal-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8506A112&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228506A112%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/size~t7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solid-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~1-1-4-14-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pinch-valves/body-material~acetal-plastic/ 2025-11-02 03:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-shelving-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screws/size~t7/ 2025-11-02 03:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454717%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15709 pages (at 49 pages/min), scraped 461 items (at 1 items/min) 2025-11-02 03:30:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/laundry-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-socket-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ax71-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-stair-treads/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~ax93/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-tags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~12-28-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/clothespins-and-spring-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-socket-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-stair-treads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ax93-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/thread-size~0-969-32-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334450%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%22437818170813%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/thread-size~4-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448881%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2649N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222649N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spanner-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000122836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=speed-control-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:30:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grounding-pipe-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spindle-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stanchions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~m26&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15760 pages (at 51 pages/min), scraped 461 items (at 0 items/min) 2025-11-02 03:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-shape~spotlight-floodlight/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2649N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222649N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/material~plastic-1/material~ptfe/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=splices/connects-to~hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333485%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scribers/tip-style~flat-chisel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=backer-rods/diameter~2-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9963492771234&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=splices&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9963492771234%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002393955%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stanchions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8496026489774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=scribers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8496026489774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002255907%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97048A746&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297048A746%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7629365453761&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=backer%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7629365453761%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000006172%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spotlight-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2649N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222649N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:31:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:31:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:31:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:31:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:31:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=77605T469&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2277605T469%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standard-high-temperature-ptfe-threaded-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seal-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bags/capacity~25-cu-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/standoffs/thread-size~m12-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:52 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-mechanisms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-retainers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/standoffs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bags/capacity-weight-~25-cu-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=guide-rails/rail-width~15-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15809 pages (at 49 pages/min), scraped 463 items (at 2 items/min) 2025-11-02 03:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stands/mount-type~slide-in/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=guide-rails/rail-width~29-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bags/capacity-weight-~25-cu-ft-/ 2025-11-02 03:32:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standoffs/thread-size~m12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-and-socket-u-joints&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/shaft-diameter~11-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=132998792534&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=guide%20rails&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B132998792534%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000068438%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stanley-10-099-knives/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standoffs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=132998792534&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=guide%20rails&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B132998792534%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163018%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/standoffs/thread-size~m12/ 2025-11-02 03:32:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stands/mount-type~slide-in&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-hones&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stanley-tape-measures/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/for-shaft-diameter~11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stanley-10-099-knives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9962802582025&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9962802582025%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010570%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=132998792534&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=guide%20rails&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B132998792534%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163018%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/for-use-with~elemental-fluorines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sprockets/for-shaft-diameter~11-16/ 2025-11-02 03:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-plunger-limit-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4256N19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224256N19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/for-use-with~methanol-methyl-alcohol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stanley-tape-measures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/star-drivers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=132998792534&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=guide%20rails&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B132998792534%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163018%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=77495A31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2277495A31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/thread-size~3-8-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984290%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/id~2-95&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15858 pages (at 49 pages/min), scraped 464 items (at 1 items/min) 2025-11-02 03:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/star-socket-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9726243375694&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=anchors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9726243375694%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7716A12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227716A12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=star-drivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/id~7-8-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506541%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-screw-nuts/thread-size~1-2-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/star-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2696N3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222696N3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/ball-screw-nuts/thread-size~1-2-5/ 2025-11-02 03:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=star-socket-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/id~7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=star-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gun-cleaning-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/formulation~buna-n&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/keyway-depth~3-3-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sprockets/id~7-8/ 2025-11-02 03:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-shanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/formulation~latex&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-tool-indicators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002132893%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer~gore-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001897965%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=angle-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002125865%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:33:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1532N41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221532N41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7649T102&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227649T102%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer~permatex&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15909 pages (at 51 pages/min), scraped 467 items (at 3 items/min) 2025-11-02 03:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=haimer-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer~gore&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer~sprayon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001897965%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ballasts/bulb-shape-number~t12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002261710%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/static-control-coveralls/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sealants/manufacturer~gore/ 2025-11-02 03:34:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002239850%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9103241651062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ballasts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9103241651062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002065540%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ballasts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6801A41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226801A41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8345K53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228345K53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/shaft-diameter~3-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001897965%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002261710%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hammer-drills/manufacturer~milwaukee-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/shaft-diameter~3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=static-control-coveralls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002271960%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/shaft-diameter~3-8-to-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000128867%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8345K53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228345K53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:34:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003178323%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002261710%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hammer-drills/manufacturer~milwaukee&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~201&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 15958 pages (at 49 pages/min), scraped 470 items (at 3 items/min) 2025-11-02 03:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hammer-drills/manufacturer~milwaukee/ 2025-11-02 03:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~27010&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6325K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226325K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002074107%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8345K53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228345K53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~3321&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-rulers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steam-separators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002082573%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6937T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226937T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002094809%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6325K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226325K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:33 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:35:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1700A251&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221700A251%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/squares/material~garolite-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steam-separators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-bar-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-bins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~352&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-butt-hinges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107700%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6325K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226325K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-precision-acme-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-rolls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=squares/material~garolite&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-bar-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74545A66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274545A66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/squares/material~garolite/ 2025-11-02 03:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16009 pages (at 51 pages/min), scraped 472 items (at 2 items/min) 2025-11-02 03:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-butt-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:13 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-butt-weld-pipe-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=banding-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stair-treads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hammers/manufacturer~milwaukee-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stamps/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standard-blower-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bar-magnets/shape~u-channel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-butt-weld-pipe-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=34195930483&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hammers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B34195930483%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002276768%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9761516107883&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stamps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9761516107883%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/steel-cup-brushes already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bar-stock/material~balsa&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3240694415818&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bar%20magnets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3240694415818%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310879%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9990257720815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bar%20stock&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9990257720815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510326%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5695K14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225695K14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-dryers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barbed-fitting-installation-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~609&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5068K111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225068K111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barbed-panel-mount-tube-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002086961%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/steel-cup-brushes already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~700&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16059 pages (at 50 pages/min), scraped 475 items (at 3 items/min) 2025-11-02 03:37:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-eye-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standard-metal-structural-tubes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002134846%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~734&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~791&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002132187%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7425A5&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227425A5%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~8029&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-eye-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stands/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 03:37:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 03:37:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002103449%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barrel-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=75825A5&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2275825A5%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002097663%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9962802582025&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stands&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9962802582025%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~ca4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7311A16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227311A16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74885A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274885A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-lockers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7425A5&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227425A5%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002115835%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-bench-tops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-machine-component-knobs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-pipe-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=75445A72&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2275445A72%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-lockers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74885A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274885A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~pc-235642&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16110 pages (at 51 pages/min), scraped 478 items (at 3 items/min) 2025-11-02 03:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7425A5&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227425A5%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handle-activated-spring-loaded-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handles/mount-type~snap-in&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-machine-component-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148764%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-pipe-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handles/thread-size~1-1-2-4-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984002022255&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=handles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984002022255%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010861%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74885A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274885A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7489A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227489A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:27 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-rivet-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984002022255&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=handles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984002022255%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:38:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handles/thread-size~10-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-safety-railings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-sheaves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=92428A710&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2292428A710%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:42 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984002022255&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=handles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984002022255%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=arbors/shank-type~round&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-rivet-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/for-use-on~lead&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4531265455181&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=arbors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4531265455181%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002129938%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-safety-railings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-sheaves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984350057510&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984350057510%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929500%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~15-5103-41500&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16161 pages (at 51 pages/min), scraped 480 items (at 2 items/min) 2025-11-02 03:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stopper-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3986N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223986N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hangers/material~foam-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002109926%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=straight-hose-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~312a-312ae&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2171N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222171N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel-tweezers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~4061&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealing-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002150132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~410076-0150&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hangers/material~foam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002082484%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002139396%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/shaft-diameter~3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71015K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271015K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~56720&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hangers/material~foam/ 2025-11-02 03:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-tweezers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71455K57&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271455K57%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strip-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2286N41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222286N41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121174%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:39:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hard-hat-area-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-shaft-diameter~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6762K48&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226762K48%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~ac675&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16213 pages (at 52 pages/min), scraped 485 items (at 5 items/min) 2025-11-02 03:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sticker-arrows/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/seals/for-shaft-diameter~3-4/ 2025-11-02 03:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~alr-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~anic0944&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133601%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~br-ccf1th&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~methane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002112401%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148474%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148179%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~cr2354&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sticker-arrows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965661%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stud-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stud-extractors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7759K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227759K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2267N51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222267N51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133601%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2286N36&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222286N36%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002088920%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7701K663&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227701K663%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrodes/aws-material-code~ercusi-a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133601%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elevator-light-bulbs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~salt-solutions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hazard-lights/bulb-type~halogen&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=supply-grilles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hazardous-location-heat-blankets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stops/mount-type~clamp-on/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrodes/aws-material-code~ercusi-a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179283%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16265 pages (at 52 pages/min), scraped 490 items (at 5 items/min) 2025-11-02 03:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5445303734503&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hazard%20lights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5445303734503%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/enamel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:12 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elevator-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994383438024&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994383438024%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002325421%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/storage-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stops/mount-type~clamp-on&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:25 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrodes/aws-material-code~ercusi-a&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/enclosures/material~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=enamel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hazardous-material-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9962004834368&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9962004834368%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010808%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/encased-magnets already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/end-caps/framing-type~press-fit/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/certification~c-ul-listed&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=storage-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=51985A67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2251985A67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/enclosure-rating~ip67&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=enclosures/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/hazardous-location-rating~nec-class-i-divisions-1-2-groups-c-d/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9241362370870&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9241362370870%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002254507%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9241362370870&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9241362370870%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000126521%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/enclosures/material~iron/ 2025-11-02 03:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-caps/framing-type~press-fit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/strain-sensors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/encased-magnets already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~mn2400&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/environmental-rating~nec-class-i-divisions-1-2-groups-c-d&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6395309901074&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6395309901074%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969330%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16309 pages (at 44 pages/min), scraped 491 items (at 1 items/min) 2025-11-02 03:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107338%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switching-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=see-through-grinding-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/switches/environmental-rating~nec-class-i-divisions-1-2-groups-c-d/ 2025-11-02 03:42:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71455K62&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271455K62%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~nic0683&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strain-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:21 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 03:42:21 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 03:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivel-forks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6395309901074&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6395309901074%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969330%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002090131%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:42:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:42:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2267N97&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222267N97%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/end-mills/mill-diameter~19-1-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivel-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/thread-size~5-16-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/end-mills/mill-diameter~2-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/thread-size~1-8-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6395309901074&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6395309901074%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969330%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=self-dumping-hoppers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swivels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333458%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swivels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334719%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~19-1mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~2-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/syringe-needles/needle-gauge~21/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:06 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16360 pages (at 51 pages/min), scraped 493 items (at 2 items/min) 2025-11-02 03:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=self-laminating-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~sp373&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/end-mills/mill-diameter~19-1mm/ 2025-11-02 03:43:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/end-mills/mill-diameter~25-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~sp395&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001728813%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/end-mills/mill-diameter~37-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099498%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002127521%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8902A56&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228902A56%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=syringe-needles/needle-gauge~21-ga&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epdm-foam-strips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=self-locking-thumb-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7604K48&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227604K48%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~25mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7604K67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227604K67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~37-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=syringes/material~borosilicate-glass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/syringe-needles/needle-gauge~21-ga/ 2025-11-02 03:43:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/end-mills/mill-diameter~25mm/ 2025-11-02 03:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001737576%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epdm-rubber-multipurpose-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1134507702014&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=syringes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1134507702014%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512599%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epdm-foam-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slotted-framing-structural-brackets/material~aluminum-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3051A41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223051A41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6500T51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226500T51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slotted-framing-structural-brackets/t-slotted-framing-rail-profile~single&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:43:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slotted-framing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-sink-compounds/material~silicone-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/voltage~9-6v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5119295949458&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=t%20slotted%20framing%20structural%20brackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5119295949458%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510631%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epdm-rubber-multipurpose-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16410 pages (at 50 pages/min), scraped 498 items (at 5 items/min) 2025-11-02 03:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5119295949458&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=t%20slotted%20framing%20structural%20brackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5119295949458%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001469376%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7864273274465&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=heat%20sink%20compounds&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7864273274465%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553774%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4844N44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224844N44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epoxies/color~safety-yellow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-acid-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epoxies/color~white/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=10405K83&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2210405K83%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2267N38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222267N38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/battery-chargers/battery-voltage~4v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=table-risers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/set-screws/military-specification~ms51963-34/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/color~safety-yellow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/strut-channel-framing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-disposal-and-recycling-containers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/color~white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-chargers/for-battery-voltage~4v&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tamper-seal-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/set-screws/military-specification~ms51963-34/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002190490%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/battery-chargers/for-battery-voltage~4v/ 2025-11-02 03:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:51 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:44:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=set-screws/military-specifications-met~ms51963-34&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7829T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227829T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strut-channel-framing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:44:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epoxies/manufacturer-model-number~dp460ns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=set-screws/specifications-met~ms51963-34&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/set-screws/military-specifications-met~ms51963-34/ 2025-11-02 03:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16458 pages (at 48 pages/min), scraped 502 items (at 4 items/min) 2025-11-02 03:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-dots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/set-screws/specifications-met~ms51963-34/ 2025-11-02 03:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/struts/strut-channel-framing-component~surface-bracket/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/struts/strut-channel-framing-component~tube/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-machines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epoxies/manufacturer-model-number~m-121hp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tapered-shims&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~dp460ns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heavy-duty-degreasers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tapping-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002150215%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=struts/strut-channel-framing-component~surface-bracket&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=struts/strut-channel-framing-component~tube&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~m-121hp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heavy-duty-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epoxies/manufacturer-model-number~mr-2000/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7467A66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227467A66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2427054843101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2427054843101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002149698%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2427054843101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2427054843101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123204%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sub-c-batteries/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heavy-duty-locksets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148188%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=33125T435&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2233125T435%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3138T3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223138T3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7370A37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227370A37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=beam-tongs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/submersible-heaters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:45:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~mr-2000&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sub-c-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16508 pages (at 50 pages/min), scraped 505 items (at 3 items/min) 2025-11-02 03:46:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121034%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=33125T435&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2233125T435%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tee-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tee-joints&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balancers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=submersible-heaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:46:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-adapter-sleeves/adapter-sleeve-trade-number~snw-08&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/for-use-with~air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121034%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=33125T435&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2233125T435%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shackles/thickness~1-2-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4305942166524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearing%20adapter%20sleeves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4305942166524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003865561%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/er70s-2-tig-welding-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-balls/material~low-carbon-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977791246039&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tees&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977791246039%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9493236633548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shackles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9493236633548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001456202%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/for-use-with~fuel-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:44 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-adapter-sleeves/adapter-sleeve-trade-number~snw-08&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 03:46:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1615424320354&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearing%20balls&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1615424320354%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511655%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121034%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:46:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-nuts/thread-size~m105&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/european-shape-hose-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977791246039&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tees&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977791246039%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=er70s-2-tig-welding-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:46:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4925004773901&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearing%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4925004773901%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452870%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1615424320354&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearing%20balls&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1615424320354%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511655%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16559 pages (at 51 pages/min), scraped 505 items (at 0 items/min) 2025-11-02 03:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-end-drivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=european-shape-hose-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/evacuation-lights already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/support-stands/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-end-hex-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/supports/mount-type~weld-on/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/exhaust-ventilators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-collar-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1615424320354&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearing%20balls&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1615424320354%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511655%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extendable-magnets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=support-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=supports/mount-type~weld-on&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/evacuation-lights already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=templates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-ventilators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tension-cables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:45 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992627010166&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992627010166%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010804%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extendable-magnets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/surge-protectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extension-cords/current~20-amp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-separators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:02 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 03:48:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 03:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extension-cords/cable-designation~rg-59-u/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=test-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16602 pages (at 43 pages/min), scraped 505 items (at 0 items/min) 2025-11-02 03:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extension-cords/cable-designation~rg-62a-u/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-seals/shaft-diameter~1-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~16003-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=surge-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/current~20a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sweat-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1157601946191&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shaft%20seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1157601946191%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000132736%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002137487%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermal-blankets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/extension-cords/current~20a/ 2025-11-02 03:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/industry-designation~rg-59-u&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/industry-designation~rg-62a-u&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rags/rag-type~sweatshirt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/extension-cords/industry-designation~rg-59-u/ 2025-11-02 03:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~1641-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sweat-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/extension-cords/industry-designation~rg-62a-u/ 2025-11-02 03:48:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002137487%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-seals/shaft-diameter~40-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114036%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermal-pads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sweatshirt-rags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-seals/shaft-diameter~45-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermocouple-assemblies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1157601946191&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shaft%20seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1157601946191%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002240223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~22208&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002137487%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1157601946191&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shaft%20seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1157601946191%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002274508%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~22308&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16654 pages (at 52 pages/min), scraped 505 items (at 0 items/min) 2025-11-02 03:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002132153%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121702%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermostats/thermostat-type~nonprogrammable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:19 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:49:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~2304-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switch-shrouds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-transfers/system-of-measurement~inch&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-006/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~2309k&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2297689035772&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thermostats&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2297689035772%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002257342%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002098200%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9042733030405&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ball%20transfers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9042733030405%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498016%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002088176%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~3015-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switch-shrouds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-006inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-cutting-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2297689035772&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thermostats&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2297689035772%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002257342%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-cutting-dies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002130171%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~3016-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~0-006inches/ 2025-11-02 03:49:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002118680%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-0156/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-valves-for-nitrogen&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/enclosure-rating~nema-6p/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~30204&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16703 pages (at 49 pages/min), scraped 505 items (at 0 items/min) 2025-11-02 03:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extensions/socket-type~ratchet-adapter/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extractors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2297689035772&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thermostats&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2297689035772%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002257342%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-locking-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002140782%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-0156inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/panel-cutout-diameter~7-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/environmental-rating~nema-6p&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extensions/socket-type~ratchet-adapter&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extractors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:50:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~0-0156inches/ 2025-11-02 03:50:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~52-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/switch-designation~3pst/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/switches/environmental-rating~nema-6p/ 2025-11-02 03:50:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9996540530177&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9996540530177%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002466413%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-052/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002150954%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/for-panel-cutout-diameter~7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-extensions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5526A2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225526A2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6004-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/industry-designation~3pst&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/switches/for-panel-cutout-diameter~7-16/ 2025-11-02 03:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-0595&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-052inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-inserts/thread-size~6-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:50:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002090349%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/switches/industry-designation~3pst/ 2025-11-02 03:51:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550225%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~0-052inches/ 2025-11-02 03:51:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/eyebolts/system-of-measurement~metric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16753 pages (at 50 pages/min), scraped 506 items (at 1 items/min) 2025-11-02 03:51:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5458250483069&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=threaded%20inserts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5458250483069%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000157155%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-lifting-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/eyebolts/thread-size~5-16-18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A301&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A301%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6200-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ball-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6206-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002132689%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyebolts/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~62302-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyebolts/thread-size~5-16-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/switching-current~9-amp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145372%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1443843511411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1443843511411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~625-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/component~ball-nut&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002134963%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6306-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1443843511411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1443843511411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333458%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110269%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7151885265529&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=threaded%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7151885265529%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002232611%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126225%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/switching-current~9a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1443843511411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1443843511411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/switching-voltage~220v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6624K66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226624K66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/switches/switching-current~9a/ 2025-11-02 03:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fabric-wall-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=banana-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6311-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fan-dampers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/wire-connection~spring-clamp-terminal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16801 pages (at 48 pages/min), scraped 507 items (at 1 items/min) 2025-11-02 03:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1443843511411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1443843511411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/switching-voltage~220v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6624K66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226624K66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/switches/wire-connection~terminal-block/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fabric-wall-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fan-dampers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9241362370870&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9241362370870%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002147981%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=three-piece-ball-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/wire-connection-type~spring-clamp-terminals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7988K6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227988K6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/switches/wire-connection-type~spring-clamp-terminals/ 2025-11-02 03:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6624K66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226624K66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fans/overall-height~11-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/wire-connection-type~terminal-block&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=through-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-197/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/switches/wire-connection-type~terminal-block/ 2025-11-02 03:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=banders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6908-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~699-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:52 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/swivels/material~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~11-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002083705%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~7210&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-197inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143072%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000158770%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/swivels/thread-size~1-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-209&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16854 pages (at 53 pages/min), scraped 508 items (at 1 items/min) 2025-11-02 03:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002084065%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~0-197inches/ 2025-11-02 03:53:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-2187&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001552493%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/swivels/material~iron/ 2025-11-02 03:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fans/overall-height~27-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/swivels/thread-size~2-1-2-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fans/overall-height~36-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/thread-size~1-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001526274%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A214&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A214%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tight-tolerance-shoulder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/swivels/thread-size~m3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454032%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A125&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A125%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barbed-food-grade-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~27-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/thread-size~2-1-2-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~36-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/swivels/thread-size~m30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000063689%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453566%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/thread-size~m3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000037998%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fans/overall-height~40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19555K54&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219555K54%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~fz-6203&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2101K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222101K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/thread-size~m30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16900 pages (at 46 pages/min), scraped 510 items (at 2 items/min) 2025-11-02 03:54:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~fz-6204&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002115948%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/timing-belts/belt-trade-number~1260xh300/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334725%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fans/overall-height~45-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002115545%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fans/overall-height~46/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070477%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/swivels/thread-type~npsm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=20375K14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2220375K14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barbed-tube-branch-tees&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~45-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~46&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barbed-tube-reducers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/timing-belts/belt-trade-number~1260xh300/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000123726%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/thread-type~npsm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000072309%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=timing-belts/belt-trade-number~770xh200&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~m88048-m88010&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19105K42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219105K42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~r12-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000151504%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7222310939460&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=timing%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7222310939460%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005438669%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:54:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002104629%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:00 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 03:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002122683%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:00 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 03:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barrel-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5845N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225845N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/syringes/dispensing-type~stream/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/drive-style~torx/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16950 pages (at 50 pages/min), scraped 514 items (at 4 items/min) 2025-11-02 03:55:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-6563&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/finish~painted/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-handle-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~1-1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tool-benches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=syringes/dispensing-type~stream&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tires&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001549886%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/drive-style~torx&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000094045%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~29-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1134507702014&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1134507702014%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002239899%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/finish~painted&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3023A256&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223023A256%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-handle-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000151397%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000012441%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/finish~zinc-plated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539034%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tool-grinders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:55:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/material~nickel-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barrier-rope&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~1-25-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/specifications-met~iso-15981/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/finish~zinc-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 16997 pages (at 47 pages/min), scraped 516 items (at 2 items/min) 2025-11-02 03:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-042&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-082&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522652%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/material~nickel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~1-55-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torch-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496338%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001492765%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/specifications-met~iso-15981&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/fasteners/material~nickel/ 2025-11-02 03:56:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:56:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001533071%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=96511A116&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2296511A116%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=96511A288&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2296511A288%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522652%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torch-tips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002098633%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95080A108&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295080A108%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/thread-size~1-3-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:40 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97447A801&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297447A801%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/totes/capacity~64-fl-oz/capacity~1-2-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torque-wrenches-for-swivel-under-load-hoist-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~1-875/diameter~1-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522652%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-slots/thread-size~10-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~1-3-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000037301%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000098629%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-2525&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455622%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498771%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:57:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/fasteners/thread-size~11-16-27/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=totes/capacity-weight-~64-fl-oz-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17051 pages (at 54 pages/min), scraped 520 items (at 4 items/min) 2025-11-02 03:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tractor-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slots/thread-size~10-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=traffic-channelizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95465A209&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295465A209%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-slotted-extrusions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:57:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/totes/capacity-weight-~64-fl-oz-/ 2025-11-02 03:57:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9950050661528&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9950050661528%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=traffic-mirrors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-slotted-framing-rollers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~2-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95465A209&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295465A209%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/fasteners/thread-size~11-16-27/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slotted-extrusions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-379&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~2-6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-5008&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~2-95-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slotted-framing-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493502%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~2-5millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001495884%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95465A209&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295465A209%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/thread-size~m140/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~2-6millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~2-5millimeters/ 2025-11-02 03:57:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 03:57:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 03:57:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~2-95millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:57:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~2-6millimeters/ 2025-11-02 03:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/table-risers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~2-95millimeters/ 2025-11-02 03:58:06 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17098 pages (at 47 pages/min), scraped 520 items (at 0 items/min) 2025-11-02 03:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-5938&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trigger-style-grease-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~m140&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~21-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/thread-size~m52/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~13a-13ae&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496670%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:58:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451255%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-6267&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001516420%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-7344&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002103786%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8491A513&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228491A513%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-couplings/fitting-connection~push-to-connect-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/take-apart-hinges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493164%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8888K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228888K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001499869%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~m52&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71015K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271015K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451255%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98042A697&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298042A697%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454977%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~3-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-couplings/connection-style~push-to-connect&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~34-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=take-apart-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71015K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271015K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:58:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451255%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tube-couplings/connection-style~push-to-connect/ 2025-11-02 03:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fastening-tape/tape-type~light-blocking/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454977%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fitting-gaskets/tube-od~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~3-875inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17147 pages (at 49 pages/min), scraped 523 items (at 3 items/min) 2025-11-02 03:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fastening-tape/width~5-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~34millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~4-9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1768933986141&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fitting%20gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1768933986141%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002214533%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~3-875inches/ 2025-11-02 03:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71015K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271015K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:17 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:59:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~34millimeters/ 2025-11-02 03:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/tape-type~light-blocking&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454977%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=43315K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2243315K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/width~5-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~4-9millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2355999507944&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2355999507944%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002248271%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/material~304-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:35 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:59:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2355999507944&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2355999507944%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000027800%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-252&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~4-9millimeters/ 2025-11-02 03:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6998A16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226998A16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-2656&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 03:59:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 03:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fdm-printer-parts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510515%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7647A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227647A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504336%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507182%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 03:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-334&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/thread-size~10-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/thread-size~7-16-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99815A029&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299815A029%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-376&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17199 pages (at 52 pages/min), scraped 526 items (at 3 items/min) 2025-11-02 04:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504336%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001494631%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000139805%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/capacity~100-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333517%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493689%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/thread-size~m22&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/thread-size~m6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89955K689&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289955K689%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feeler-gauges/thicknesses-included~0-002-to-0-01-0-001-increments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334755%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000148470%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504336%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity-weight-~100-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5148K711&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225148K711%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89955K689&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289955K689%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tanks/capacity-weight-~100-gal-/ 2025-11-02 04:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/tube-od~125-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~0-002-to-0-01-0-001-increments-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/tube-od~29-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~6-25-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:46 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/capacity~2-1-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002218454%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/feeler-gauges/thickness~0-002-to-0-01-0-001-increments-/ 2025-11-02 04:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feeler-gauges/thicknesses-included~0-004-to-0-025-0-001-increments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002215454%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000124114%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feeler-gauges/thickness~0-011/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~6-3-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89955K689&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289955K689%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3626N17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223626N17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/capacity~3-3-8-qt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6963N75&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226963N75%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000104043%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17248 pages (at 49 pages/min), scraped 528 items (at 2 items/min) 2025-11-02 04:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity-weight-~2-1-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~0-004-to-0-025-0-001-increments-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~0-011&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tanks/capacity-weight-~2-1-gal-/ 2025-11-02 04:01:19 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity-weight-~3-3-8-qt-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/feeler-gauges/thickness~0-004-to-0-025-0-001-increments-/ 2025-11-02 04:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~7-2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-626&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8264408215759&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8264408215759%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001461603%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tanks/capacity-weight-~3-3-8-qt-/ 2025-11-02 04:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/capacity~26-qt/capacity~6-5-gal/capacity~6-1-2-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~7-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~7-4-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/capacity~6-3-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496851%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2083A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222083A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~7-6-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~7-2millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001515727%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001555456%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity-weight-~6-1-2-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~7-5millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~7-2millimeters/ 2025-11-02 04:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A352&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A352%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/capacity~70-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity-weight-~6-3-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feeler-gauges/thicknesses-included~0-075/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:01:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tanks/capacity-weight-~6-1-2-gal-/ 2025-11-02 04:01:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~7-5millimeters/ 2025-11-02 04:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feeler-gauges/thickness~0-3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tanks/capacity-weight-~6-3-gal-/ 2025-11-02 04:02:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feeler-gauges/thickness~0-4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001555456%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/for-use-with~acetylene-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17295 pages (at 47 pages/min), scraped 532 items (at 4 items/min) 2025-11-02 04:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~9-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity-weight-~70-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feet/shape~tapered-round/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~0-075&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~0-3mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tanks/capacity-weight-~70-gal-/ 2025-11-02 04:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8264408215759&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8264408215759%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001454256%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~0-4mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001555456%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/empty-acetylene-tanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~9-5inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/feeler-gauges/thickness~0-3mm/ 2025-11-02 04:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feet/shape~tapered-square/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feet/shape~tapered-round&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/feeler-gauges/thickness~0-4mm/ 2025-11-02 04:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%226406895909995%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/diameter~9-5inches/ 2025-11-02 04:02:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/material~nickel-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8031429930803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8031429930803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310817%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8264408215759&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8264408215759%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001454256%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:46 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~13-032-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7986A41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227986A41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feet/shape~tapered-square&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~11-9997-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:54 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feet/shape~tapered-round&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:02:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:02:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/for-use-with~deionized-water-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8031429930803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8031429930803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002308910%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/material~nickel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fencing/fencing-component~fencing-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001499664%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8264408215759&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8264408215759%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001454256%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967804%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~13-032mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17342 pages (at 47 pages/min), scraped 533 items (at 1 items/min) 2025-11-02 04:03:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/thread-size~1-1-8-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:09 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feet/shape~tapered-square&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:03:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shafts/material~nickel/ 2025-11-02 04:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/id~13-032mm/ 2025-11-02 04:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334742%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tape-insulation/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fencing/component~privacy-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fencing/component~fencing-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:24 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fep-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/fencing/component~privacy-strips/ 2025-11-02 04:03:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:03:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/fencing/component~fencing-ties/ 2025-11-02 04:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fep-plastic/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing-management&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing-routers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tape-measures/length~5-m/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-insulation&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:03:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fep-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ferrules/for-use-with~pharmaceuticals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/taps/for-use-on~hardened-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fep-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ferrules/length~11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/length~5-m&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/squares/material~uhmw-polyethylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/for-use-with~pharmaceuticals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17386 pages (at 44 pages/min), scraped 533 items (at 0 items/min) 2025-11-02 04:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~24-9997-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5504631511088&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5504631511088%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001614823%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/for-use-on~hardened-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shears/for-use-on~iron-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/length~11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968243%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498641%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:18 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/length~5-m&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ferrules/specifications-met~astm-standard/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=uhmw-polyethylene-squares&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929763%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5847281553902&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shears&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5847281553902%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929691%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001604843%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4759K219&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224759K219%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~25-025-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498641%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7195N16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227195N16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=unistrut-pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/specifications-met~astm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fh-024-6-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001494765%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~28-076-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sheave-alignment-lasers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/ferrules/specifications-met~astm/ 2025-11-02 04:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498641%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/taps/thread-size~1-4-56/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7195N16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227195N16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fh024-6-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~28-076mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498330%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17434 pages (at 48 pages/min), scraped 534 items (at 1 items/min) 2025-11-02 04:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/taps/thread-size~1-8-27/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~32-01-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/id~28-076mm/ 2025-11-02 04:05:17 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~1-4-56&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7195N16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227195N16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001502482%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/taps/thread-size~3-16-100-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fiberglass-boards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~35-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/taps/thread-size~5-16-18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454282%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~1-8-27&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001495920%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~276-h3m-15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455337%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:41 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:05:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~3-16-100&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-boards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005296092%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~5-16-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aspirators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/taps/thread-size~3-16-100/ 2025-11-02 04:05:51 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~375-h5m-25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7939K526&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227939K526%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333458%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-grooves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shelving-angle-brackets/material~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005297315%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~55-06-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17484 pages (at 50 pages/min), scraped 535 items (at 1 items/min) 2025-11-02 04:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-aspirators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shelving/component~stacking-cap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507865%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/taps/thread-size~m8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shelving-angle-brackets/material~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8128528462812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shelving&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8128528462812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002235553%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005297315%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-instruments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shelving-angle-brackets/material~stainless-steel/ 2025-11-02 04:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~6-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-lids&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8815N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228815N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~m8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005297315%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shielded-cable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333499%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shields/barrier-type~enclosure&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/taps/thread-type~nptf/pipe-size~1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~6-500mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8815N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228815N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-tube-reducers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9956488056472&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shields&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9956488056472%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002278513%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/id~6-500mm/ 2025-11-02 04:06:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:06:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valve-heaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:06:59 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shields/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-type~nptf/pipe-size~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:03 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shields/barrier-type~enclosure&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8815N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228815N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17536 pages (at 52 pages/min), scraped 535 items (at 0 items/min) 2025-11-02 04:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~9-300-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~8-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filter-elements/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/military-specification~ms17795-141/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000067115%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/for-use-with~ethylene-glycol-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~9-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shields/material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505283%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:24 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-type~nptf/pipe-size~1-4&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968988%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:25 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001499693%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~9-300mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shields/material~steel/ 2025-11-02 04:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filter-elements&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/military-specifications-met~ms17795-141&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tee-nuts/thread-size~5-16-24-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filters/for-use-with~beverage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8486A465&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228486A465%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/id~9-300mm/ 2025-11-02 04:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:40 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:07:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/military-specifications-met~ms17795-141/ 2025-11-02 04:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shims/material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tee-nuts/thread-size~m22/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:07:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/unthreaded-spacers/material~polypropylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/material~leather-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tees/fitting-connection~threaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tee-nuts/thread-size~5-16-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-344&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/for-use-with~beverage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:07:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shims&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000067117%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tee-nuts/thread-size~5-16-24/ 2025-11-02 04:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000143482%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967788%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tee-nuts/thread-size~m22&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17582 pages (at 46 pages/min), scraped 536 items (at 1 items/min) 2025-11-02 04:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/polypropylene-unthreaded-spacers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shims/shape~round/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/connection-style~threaded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3487712981410&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3487712981410%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334755%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shims/polypropylene-unthreaded-spacers/ 2025-11-02 04:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shim-stock/sold-as~assortment/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tees/connection-style~threaded/ 2025-11-02 04:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tees/for-use-with~propylene-glycol/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vga-cables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:24 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tee-nuts/thread-size~m22&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tees/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tees/material~copper-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/shape~ring-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tees/material~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/shim-stock-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shims/shape~ring-1/ 2025-11-02 04:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/for-use-with~propylene-glycol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vibration-damping-mats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/material~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/material~copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shims/shim-stock-sets/ 2025-11-02 04:08:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/thickness~0-021&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/material~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977791246039&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977791246039%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967516%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-315&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tees/material~brass/ 2025-11-02 04:08:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tees/material~copper/ 2025-11-02 04:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fine-filtration-vacuums/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shims&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001458327%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:08:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tees/material~fiberglass/ 2025-11-02 04:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3319N34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223319N34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/thickness~0-09&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000168132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shims/thickness~0-900-mm/thickness~0-9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17628 pages (at 46 pages/min), scraped 536 items (at 0 items/min) 2025-11-02 04:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shims&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001467155%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fine-filtration-vacuums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wall-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-878&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3319N34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223319N34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-8135&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000168132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/thickness~0-9mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000019925%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000084870%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wall-dividers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shims/thickness~0-9mm/ 2025-11-02 04:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3319N34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223319N34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:42 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000168132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/thickness~5-8-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~11-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~12-018-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/finish~phosphate-coated/color~black-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shims&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001455259%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000121759%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000080647%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~13-03-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:09:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washer-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~136-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=18635A21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2218635A21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17681 pages (at 53 pages/min), scraped 536 items (at 0 items/min) 2025-11-02 04:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000152210%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/finish~black-phosphate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/terminals/wire-connection-method~compression/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flanges/flange-od~10-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/screw-size~no-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~15-88-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/finish~black-phosphate/ 2025-11-02 04:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flanges/flange-od~13-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~16-018-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~136-000-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334338%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=terminals/wire-connection-type~compression&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/flange-od~10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000125427%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~15-88mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-removal-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/flange-od~13-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/terminals/wire-connection-type~compression/ 2025-11-02 04:10:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/flanges/flange-od~10/ 2025-11-02 04:10:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~16-018mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/od~15-88mm/ 2025-11-02 04:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flanges/flange-od~4-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000039642%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/od~16-018mm/ 2025-11-02 04:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000125427%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flanges/material~copper-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=44685K175&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2244685K175%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shock-absorbing-casters/weight-capacity-per-caster~2000-lb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-resistant-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/flange-od~4-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wear-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17728 pages (at 47 pages/min), scraped 538 items (at 2 items/min) 2025-11-02 04:11:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weatherstripping&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000125427%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119897%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/material~copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shock-absorbing-casters-(capacity-700-to-2%2c000-lbs)&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/flanges/material~copper/ 2025-11-02 04:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=44685K223&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2244685K223%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=web-strap-pullers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shop-coats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flare-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=webbing-straps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~2-501&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thermometers/mount-type~adhesive-back/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:42 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~2-717&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shop-vac-vacuum-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~20-03-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000084662%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flare-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000025151%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-curtains&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:11:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-electrodes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermometers/mount-type~adhesive&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~22-04-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shoulder-bolts/thread-size~5-8-11-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:12:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:12:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~20-03mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17779 pages (at 51 pages/min), scraped 539 items (at 1 items/min) 2025-11-02 04:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flat-head-drywall-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6790060835426&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6790060835426%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010482%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000098386%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thread/finish~phosphate-coated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/od~20-03mm/ 2025-11-02 04:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shoulder-screws/military-specification~ms51576-38/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shoulder-screws/military-specification~ms51575-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-torches/torch-series-number~18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shoulder-bolts/thread-size~5-8-11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flat-head-drywall-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flexible-pipe-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~29-04-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7140856590719&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=welding%20torches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7140856590719%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002078403%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread/finish~phosphate-coated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shoulder-bolts/thread-size~5-8-11/ 2025-11-02 04:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shoulder-screws/military-specifications-met~ms51576-38&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shoulder-screws/military-specifications-met~ms51575-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:41 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-torches/torch-series-number~18&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:12:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3913099702390&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3913099702390%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001555313%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shoulder-screws/military-specifications-met~ms51576-38/ 2025-11-02 04:12:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shoulder-screws/military-specifications-met~ms51575-3/ 2025-11-02 04:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flexible-pipe-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/floats/for-use-with~hydrochloric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~29-04mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~30-04-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9666K142&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229666K142%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~3-253&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:12:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/od~29-04mm/ 2025-11-02 04:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/floor-cleaners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/floats/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shunt-carbon-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floats/for-use-with~hydrochloric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17822 pages (at 43 pages/min), scraped 540 items (at 1 items/min) 2025-11-02 04:13:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~30-04mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/floor-posts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2394922483006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2394922483006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002963068%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000042606%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-rivets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floats/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/od~30-04mm/ 2025-11-02 04:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3748K31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223748K31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/floats/material~aluminum/ 2025-11-02 04:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-rod-vibration-dampers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~38-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-rods/material~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-posts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rivets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000112362%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:13:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sight-glass-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rod-vibration-dampers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flow-controls/for-use-with~beverage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flow-controls/for-use-with~citric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flow-controls/for-use-with~helium/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/threaded-rods/material~iron/ 2025-11-02 04:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-rods/thread-size~1-4-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flow-controls/for-use-with~soap-solutions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sign-boards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-controls/for-use-with~beverage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-controls/for-use-with~citric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-controls/for-use-with~helium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17867 pages (at 45 pages/min), scraped 541 items (at 1 items/min) 2025-11-02 04:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9072876622555&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9072876622555%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967788%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9072876622555&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9072876622555%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967437%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~4mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9072876622555&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9072876622555%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964764%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/thread-size~1-4-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~32-051-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/threaded-rods/thread-size~1-7-8-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/od~4mm/ 2025-11-02 04:14:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-rods/thread-size~2-1-2-4-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7151885265529&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7151885265529%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453967%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493017%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sjo-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sleeves/performance~chemical-resistant-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-rods/thread-size~2-1-4-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A703&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A703%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=signs/accident-prevention-message-type~slip-and-fall-prevention&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sleeves/material~titanium-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/thread-size~2-1-2-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flow-valves/valve-type~ball/for-use-with~citric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/threaded-rods/thread-size~1-7-8-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5161877211177&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=signs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5161877211177%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000164068%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sleeves/performance-properties~chemical-resistant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/thread-size~2-1-4-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=299320586602&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sleeves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B299320586602%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000097208%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/threaded-rods/thread-size~2-1-2-4/ 2025-11-02 04:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slider-bed-conveyors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sleeves/performance-properties~chemical-resistant/ 2025-11-02 04:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flow-valves/valve-type~ball/for-use-with~soap-solutions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7151885265529&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7151885265529%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455638%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:14:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~76-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-valves/valve-type~ball/for-use-with~citric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-stops/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flowmeters/for-use-with~hydrogen/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=94815A125&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2294815A125%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17913 pages (at 46 pages/min), scraped 542 items (at 1 items/min) 2025-11-02 04:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flowmeters/for-use-with~inert-gas/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=339191585183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B339191585183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247047%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967437%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=socket-head-cap-screws/thread-size~1-2-13&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inline-ball-joint-linkages/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-valves/valve-type~ball/for-use-with~soap-solutions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3981476179935&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=socket%20head%20cap%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3981476179935%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-stops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=339191585183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B339191585183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247047%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965378%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flowmeters/for-use-with~hydrogen&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=socket-head-cap-screws/thread-size~6-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flowmeters/for-use-with~inert-gas&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/voltage-testers/manufacturer~fluke/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~76mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/rod-ends-and-linkages/inline-ball-joint-linkages-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=453523490803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B453523490803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967523%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3981476179935&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=socket%20head%20cap%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3981476179935%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000157155%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=453523490803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B453523490803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967870%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/od~76mm/ 2025-11-02 04:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fmc-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=78595A499&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2278595A499%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4218K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224218K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fluke-voltage-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=soft-point-socket-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=belt-tension-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tie-down-winches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/foam-bar-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fmc-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bench-tops/material~laminate/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/for-housing-id~17-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/foil-raw-materials/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 17959 pages (at 46 pages/min), scraped 545 items (at 3 items/min) 2025-11-02 04:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tie-down-winches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-bar-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=single-edge-razor-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-housing-id~2-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ties/length~13-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bench-tops/material~plastic-laminate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-housing-id~17mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foil-raw-materials&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/folding-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000017604%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/for-housing-id~55-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bench-tops/material~plastic-laminate/ 2025-11-02 04:16:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/spacers/for-housing-id~17mm/ 2025-11-02 04:16:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:16:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sink-drain-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~13-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=beverages&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/screw-size~m28/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ties/overall-length~36-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=folding-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6635333760770&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6635333760770%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001618159%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-housing-id~55mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bicycle-sprockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sit-stand-workstations&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-cabinets/environment~food-and-beverage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:16:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/spacers/for-housing-id~55mm/ 2025-11-02 04:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/shaft-diameter~1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-screw-size~m28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~36-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/shaft-diameter~4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18006 pages (at 47 pages/min), scraped 545 items (at 0 items/min) 2025-11-02 04:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/spacers/for-screw-size~m28/ 2025-11-02 04:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6635333760770&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6635333760770%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001630102%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/food-industry-seals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-industry-electrical-cabinets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-shaft-diameter~1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=skate-wheels/wheel-diameter~3-5-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bird-repellents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3097T44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223097T44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-shaft-diameter~4mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bit-braces&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/spacers/for-shaft-diameter~1-8/ 2025-11-02 04:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8350698174345&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=skate%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8350698174345%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001939329%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~33-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/spacers/for-shaft-diameter~4mm/ 2025-11-02 04:17:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:17:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-industry-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000126026%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/force-pressure-sensors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2289T515&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222289T515%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/screw-size~m10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/foot-switch-pedals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/screw-size~m12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/screw-size~m135&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/screw-size~m2-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~6-3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334357%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tip-indicators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334339%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455667%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334803%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000156434%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=force-pressure-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18058 pages (at 52 pages/min), scraped 547 items (at 2 items/min) 2025-11-02 04:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foot-switch-pedals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blades/precision-knife-blade-type~tapered-tip-for-carving&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334357%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tip-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/leveling-washers/washer-type~tapered/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-clamping-hangers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tires/tire-size~2-80-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334339%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982658580091&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=blades&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982658580091%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003034555%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/framing/shape~round-tube/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blasting-glass-beads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=38995A77&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2238995A77%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334357%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing/leveling-washers-for-i-beams-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tires/tire-size~2-80-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334339%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing/shape~round-tube&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/framing/leveling-washers-for-i-beams-6/ 2025-11-02 04:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=split-rivets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/titanium-discs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1409080599349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1409080599349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099630%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/free-machining-brass/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971113029698&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971113029698%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002311201%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fuel-line-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=53795T22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2253795T22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:19:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-paint/color~beige&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18105 pages (at 47 pages/min), scraped 548 items (at 1 items/min) 2025-11-02 04:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=titanium-discs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=free-machining-brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5212847902408&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spray%20paint&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5212847902408%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002203687%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blue-dyed-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fume-cleaners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuel-line-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blueprint-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sling-guards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7719T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227719T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/furniture-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bluing-ink&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fume-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-loaded-seals/for-use-with~phosphoric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slip-joint-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/furniture-pads/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=boiler-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9759667665353&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spring%20loaded%20seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9759667665353%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965403%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=boiler-level-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/boiler-cleaners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=boiler-pressure-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=furniture-pads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/european-fuses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18152 pages (at 47 pages/min), scraped 550 items (at 2 items/min) 2025-11-02 04:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bollard-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slit-corrugated-sleeving&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fuses/manufacturer-equivalent-fuse~frn-r-id/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolt-cutters/for-use-on~medium-hard-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tool-mounts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~11-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7394355010619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolt%20cutters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7394355010619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930103%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/boiler-cleaners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fuses/manufacturer-equivalent-fuse~trs-r-id/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuses/european-fuses-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/military-specification~ms51575-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2891885695101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=springs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2891885695101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001591806%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuses/manufacturer-equivalent~frn-r-id&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3771A161&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223771A161%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8931638457760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8931638457760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%225814370497652%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/material~fiberglass-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tool-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/fuses/manufacturer-equivalent~frn-r-id/ 2025-11-02 04:20:42 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuses/manufacturer-equivalent~trs-r-id&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003423545%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:48 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/military-specifications-met~ms51575-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/military-specification~ms51575-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/fuses/manufacturer-equivalent~trs-r-id/ 2025-11-02 04:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/galvanized-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8931638457760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8931638457760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%225814370497652%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bolts/military-specifications-met~ms51575-24/ 2025-11-02 04:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/galvanized-eyebolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tooling-plates/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=small-batch-mixers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18196 pages (at 44 pages/min), scraped 551 items (at 1 items/min) 2025-11-02 04:21:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/military-specifications-met~ms51575-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=galvanized-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-size~1-3-4-8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8931638457760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8931638457760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%225814370497652%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=galvanized-eyebolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tooling-plates/material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~1-1-8-5-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bolts/military-specifications-met~ms51575-4/ 2025-11-02 04:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~3inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~1-3-4-12-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tooling-plates/material~steel/ 2025-11-02 04:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452028%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/springs/length~3inches/ 2025-11-02 04:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453574%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~1-3-4-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:33 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/garden-hose-shut-off-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~6-3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~1-5-8-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=smoke-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A757&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A757%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bolts/thread-size~1-3-4-8/ 2025-11-02 04:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~79-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454641%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~6-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/torque-clutches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=garden-hose-shut-off-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~6-3millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/garolite/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:21:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gas-piston-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2891885695101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=springs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2891885695101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001630761%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~6millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~79-5millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/springs/length~6-3millimeters/ 2025-11-02 04:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454641%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18244 pages (at 48 pages/min), scraped 552 items (at 1 items/min) 2025-11-02 04:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torque-clutches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/springs/length~6millimeters/ 2025-11-02 04:22:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/springs/length~79-5millimeters/ 2025-11-02 04:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-piston-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=garolite&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/torx-drivers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2891885695101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=springs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2891885695101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001630761%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/torx-screwdriver-bit-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/torx-thread-forming-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=snap-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bonding-and-grounding-reels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454641%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tote-box-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torx-drivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2891885695101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=springs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2891885695101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001630761%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torx-screwdriver-bit-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torx-thread-forming-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/totes/capacity~10-1-4-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:47 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=snap-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gaskets/for-use-with~chlorine/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bore-gauges/manufacturer-model-number~173mat&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tote-box-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gaskets/for-use-with~coolant/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/totes/capacity~12-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=totes/capacity-weight-~10-1-4-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=191095061359&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bore%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B191095061359%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079958%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18290 pages (at 46 pages/min), scraped 552 items (at 0 items/min) 2025-11-02 04:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gaskets/for-use-with~food/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=soap-cartridges/manufacturer~stoko&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/totes/capacity-weight-~10-1-4-gal-/ 2025-11-02 04:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/for-use-with~chlorine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2070A44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222070A44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/for-use-with~coolant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=totes/capacity-weight-~12-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/soap-cartridges/manufacturer~stoko/ 2025-11-02 04:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965544%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/for-use-with~food&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bore-gauges/manufacturer-model-number~829b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969065%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/totes/capacity-weight-~12-gal-/ 2025-11-02 04:23:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:23:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bore-liners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=soapstone-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/totes/material~cardboard/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=77495A21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2277495A21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/shaft-diameter~6-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967679%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=191095061359&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bore%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B191095061359%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145510%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000145054%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2100A2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222100A2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969065%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gaskets/thickness~0-03/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gaskets/thickness~0-218/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=totes/material~cardboard&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/traceable-rulers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:23:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-51&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8786789701969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8786789701969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511131%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000145054%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-8-2/id~0-125&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18338 pages (at 48 pages/min), scraped 555 items (at 3 items/min) 2025-11-02 04:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969065%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497459%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/thickness~0-03&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/thickness~0-218&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001504455%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001494745%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=traceable-rulers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-duct&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001459026%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/transfer-balls/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001463894%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000145054%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2750K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222750K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bottles/capacity~3-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/transformers/voltage~4-5v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/for-use-with~humidifiers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-enclosures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=transfer-balls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:45 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=socket-wrench-adapters/square-drive-size~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bottles/contents-message~ethyl-acetate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bottles/capacity-weight-~3-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=transformers/voltage~4-5v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/for-use-with~humidifiers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1687313801023&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=socket%20wrench%20adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1687313801023%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163995%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/trash-cans/capacity~21-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7034539236718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bottles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7034539236718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002132952%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bottles/capacity-weight-~3-ml/ 2025-11-02 04:25:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=803742726147&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B803742726147%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553720%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bottles/contents-message~solvent-wash-bottle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=soda-ash&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967760%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5523A38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225523A38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18390 pages (at 52 pages/min), scraped 556 items (at 1 items/min) 2025-11-02 04:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=40185T42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2240185T42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70235K26&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270235K26%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7034539236718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bottles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7034539236718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002084950%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2123K18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222123K18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bow-saws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-open-end-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=43555T12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2243555T12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trash-cans/capacity-weight-~21-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/trash-cans/capacity~95-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-pipe-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/trash-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:25:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/trash-cans/capacity-weight-~21-gal-/ 2025-11-02 04:25:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-pipe-elbows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/measurement-unit~hectopascals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trash-cans/capacity-weight-~95-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/thread-type~unc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-shim-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solder-end-globe-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-quick-release-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solder-flux-remover&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trash-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/trash-cans/capacity-weight-~95-gal-/ 2025-11-02 04:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-strut-channel-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/scale~hpa&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gears/material~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18438 pages (at 48 pages/min), scraped 561 items (at 5 items/min) 2025-11-02 04:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/thread-type~unc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gauges/scale~hpa/ 2025-11-02 04:26:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glass/material~ceramic-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002102665%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-workbenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-rectangular-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standard-galvanized-grade-dh-steel-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gears/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-thin-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standoffs/thread-size~1-1-4-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solenoids/body-material~zinc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gears/material~iron/ 2025-11-02 04:26:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass/material~ceramic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-shoulder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-drill-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4728407233061&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=standoffs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4728407233061%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-end-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=168950821422&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=solenoids&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B168950821422%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001573916%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/glass/material~ceramic/ 2025-11-02 04:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~3-000-mm/id~3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standoffs/thread-size~5-16-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4077N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224077N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=break-bells&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-fittings/tube-od~2-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4728407233061&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=standoffs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4728407233061%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334815%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-drill-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-fittings/tube-od~3-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-end-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~3mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18486 pages (at 48 pages/min), scraped 562 items (at 1 items/min) 2025-11-02 04:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solenoids/voltage~6v-dc-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-fittings/tube-od~38-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/breakers/current~120-amp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/id~3mm/ 2025-11-02 04:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-tube-od~2-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=168950821422&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=solenoids&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B168950821422%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553639%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4728407233061&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=standoffs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4728407233061%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334815%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gloves/color~fluorescent-green/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-tube-od~3-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tube-fittings/for-tube-od~2-1-4/ 2025-11-02 04:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gloves/color~red-fluorescent-yellow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4916K2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224916K2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tube-fittings/for-tube-od~3-1-4/ 2025-11-02 04:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-tube-od~38mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=breakers/current~120a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-acorn-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-fittings/for-use-with~air/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4728407233061&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=standoffs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4728407233061%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334815%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gloves/color~white-gray/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/color~fluorescent-lime-green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tube-fittings/for-tube-od~38mm/ 2025-11-02 04:27:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/breakers/current~120a/ 2025-11-02 04:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/color~red-fluorescent-yellow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4916K2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224916K2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gloves/color~fluorescent-lime-green/ 2025-11-02 04:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gloves/color~white/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gloves/for-use-with~turpentine/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002198532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-use-with~air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:27:59 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/color~white-gray&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solvents/for-use-on~bronze-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9353T72&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229353T72%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4916K2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224916K2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18533 pages (at 47 pages/min), scraped 562 items (at 0 items/min) 2025-11-02 04:28:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/for-use-on~grout/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002192133%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/color~white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/for-use-with~turpentine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-globe-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=966787406552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=solvents&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B966787406552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929994%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9465T8&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229465T8%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967593%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9353T72&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229353T72%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/for-use-on~grout&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-racks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929998%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-manifolds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-pipe-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7770A27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227770A27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sound-level-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-precision-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9353T72&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229353T72%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-plug-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:28:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brother-tc20-label-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~14270/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18583 pages (at 50 pages/min), scraped 564 items (at 2 items/min) 2025-11-02 04:29:09 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:29:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:13 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:29:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~25/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brush-seaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~3106/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brushless-motors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~14270&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bt-truck-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bubble-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/lubricant~sae-30-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143902%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/for-use-with~solvents/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002052616%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~310a-b/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~3106&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=66215A31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2266215A31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/id~0-014/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002091775%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~3922/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002137733%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7468A62&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227468A62%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buckets/capacity~6-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/for-use-with~solvents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:29:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~310a-b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74545A22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274545A22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~0-014&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/id~0-236/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~3922&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179285%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002140547%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18630 pages (at 47 pages/min), scraped 567 items (at 3 items/min) 2025-11-02 04:30:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001494902%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buckets/capacity~2-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002122642%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity-weight-~6-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=66195A13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2266195A13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/id~0-43/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5560K623&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225560K623%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~0-236&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stoppers/shape~square&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1700A259&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221700A259%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/buckets/capacity-weight-~6-gal-/ 2025-11-02 04:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493401%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977840261548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stoppers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977840261548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310398%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity-weight-~2-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~4851/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~50133/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~0-43&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2153T118&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222153T118%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buffing-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/buckets/capacity-weight-~2-gal-/ 2025-11-02 04:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1700A259&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221700A259%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498799%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~505-uv/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/id~1-4-2/id~0-25/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/id~1-65/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~4851&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~50133&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/id~1-8-2/id~0-125/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~560/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1700A259&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221700A259%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002120900%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/screw-size~no-14/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002083616%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18675 pages (at 45 pages/min), scraped 570 items (at 3 items/min) 2025-11-02 04:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~505-uv&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~1-65&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74795A73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274795A73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7605A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227605A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002098982%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~560&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tubing/id~1-4/ 2025-11-02 04:31:21 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001500858%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-screw-size~no-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tubing/id~1-8/ 2025-11-02 04:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6937T35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226937T35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/id~18-000-mm/id~18-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~587/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002096530%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2108T77&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222108T77%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/spacers/for-screw-size~no-14/ 2025-11-02 04:31:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~1-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74795A73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274795A73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6937T769&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226937T769%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=straps/material~stainless-steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~18mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~587&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stretch-wrap/film-gauge~90&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:31:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bulkheads/for-use-with~liquid-helium-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74795A73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274795A73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9733419879211&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=straps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9733419879211%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/tubing/id~18mm/ 2025-11-02 04:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002103006%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulkheads/for-use-with~hydrochloric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8246808234932&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stretch%20wrap&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8246808234932%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003016205%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000040115%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulkheads/for-use-with~isopropyl-alcohol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18724 pages (at 49 pages/min), scraped 574 items (at 4 items/min) 2025-11-02 04:32:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/id~3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9963946302057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bulkheads&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9963946302057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002963068%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:32:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9963946302057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bulkheads&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9963946302057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965701%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulkheads/for-use-with~liquid-helium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:19 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:32:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~74/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1282K42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221282K42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~87937/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50715K124&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250715K124%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bulkheads/for-use-with~liquid-helium/ 2025-11-02 04:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulkheads/thread-type~bspp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulkheads/thread-type~npsm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/od~1-520/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498330%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~74&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9963946302057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bulkheads&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9963946302057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000037076%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/enclosure-rating~nema-4x&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~87937&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9963946302057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bulkheads&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9963946302057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000151504%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002125070%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bumpers/thread-size~m24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50785K103&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250785K103%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9241362370870&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9241362370870%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000093762%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002088716%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5463K81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225463K81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/od~1-520&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/switching-current~15-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7610A36&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227610A36%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7150684641693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bumpers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7150684641693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334859%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055597%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9241362370870&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9241362370870%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001618842%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18775 pages (at 51 pages/min), scraped 579 items (at 5 items/min) 2025-11-02 04:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bundyweld-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/od~12-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/od~17-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3994K15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223994K15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/od~2-1-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bung-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/shape~tapered-rectangle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/manufacturer-model-number~rtv106/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/od~12-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/od~17-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310325%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3994K15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223994K15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/od~2-1-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000135242%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushing-bore-hubs/bushing-trade-number~1615&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000123228%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/switching-voltage~115v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/thread-size~m260&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:42 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/shape~tapered-rectangle&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~rtv106&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000157115%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1242777880953&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushing%20bore%20hubs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1242777880953%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001996131%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=52945K48&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2252945K48%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9241362370870&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9241362370870%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001632957%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=52375K22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2252375K22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454427%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grade-1144-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3994K15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223994K15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123054%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:33:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9118T5&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229118T5%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7095N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227095N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74975A53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274975A53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18825 pages (at 50 pages/min), scraped 582 items (at 3 items/min) 2025-11-02 04:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grade-2h-steel-coupling-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/thread-size~3-137-12-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/steel/material~steel-2/material~52100-alloy-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454427%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:13 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:34:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-1144-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~1-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grade-9-titanium/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grade-b6-stainless-steel-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-2h-steel-coupling-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454032%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~3-137-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-52100-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~5-8-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454427%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95365A131&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295365A131%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bushings/thread-size~3-137-12/ 2025-11-02 04:34:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grade-c-steel-locknuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455614%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-9-titanium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-b6-stainless-steel-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tabs/material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/take-up-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95365A513&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295365A513%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:34:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:34:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:34:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grade-f-steel-locknuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-c-steel-locknuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t8-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/turnbuckles/thread-size~3-8-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18868 pages (at 43 pages/min), scraped 586 items (at 4 items/min) 2025-11-02 04:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tabs/material~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=take-up-conveyor-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tamper-resistant-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tamper-resistant-torx-drivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tabs/material~plastic/ 2025-11-02 04:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-f-steel-locknuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bussmann-type-kts-r-fuses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=turnbuckles/thread-size~3-8-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/graphene-rubber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=butane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grate-decking/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/length~40-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3217042704133&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3217042704133%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/length~6-ft/length~72&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/two-part-sealants/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5504631511088&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tape%20measures&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5504631511088%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001612975%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=graphene-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spill-pans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5504631511088&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tape%20measures&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5504631511088%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001618631%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001628228%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spindle-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grate-decking&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:53 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/length~40-ft&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:35:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:35:57 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/length~6-ft/length~72&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-type~npt/pipe-size~1-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=two-part-sealants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tygon-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spiral-duct-dampers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18919 pages (at 51 pages/min), scraped 586 items (at 0 items/min) 2025-11-02 04:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000074440%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease/trade-name~krytox-xht-bdx/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bx46-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/for-use-with~nitric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-type~npt/pipe-size~1-1-4&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/pipe-size~3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tees/specifications-met~asme-standard/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tygon-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tees/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977791246039&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tees&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977791246039%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965017%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977791246039&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tees&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977791246039%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119922%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~krytox-xht-bdx&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/pipe-size~5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5306N22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225306N22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002140084%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bypass-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977791246039&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tees&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977791246039%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000046129%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~3-8-2/od~0-375/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/specifications-met~asme&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=c-clamp-pads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tees/material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-shape-number~c7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=10195K22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2210195K22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4881K013&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224881K013%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tees/specifications-met~asme/ 2025-11-02 04:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease/trade-name~lu-700-food-grade-machinery/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tees/material~steel/ 2025-11-02 04:36:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~c158/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/u-bolts/material~rubber-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=c-shape-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/u-brackets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=terminals/wire-connection-method~solder/wire-connection-method~clamp-on&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/od~3-8/ 2025-11-02 04:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 18967 pages (at 48 pages/min), scraped 589 items (at 3 items/min) 2025-11-02 04:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~lu-700-food-grade-machinery&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=split-head-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9784009411319&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=terminals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9784009411319%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002464574%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003070703%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/u-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=u-bolts/u-bolts-with-mounting-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=c158-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=u-bolts/material~rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002141512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=u-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/u-bolts/u-bolts-with-mounting-plates/ 2025-11-02 04:37:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/u-bolts/material~rubber/ 2025-11-02 04:37:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1155K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221155K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~molykote-557-silicone-dry-film/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread/material~stainless-steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease/trade-name~molykote-food-machinery-spray-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=u-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cabinets/mounting-location~enclosure-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-rods/material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3913099702390&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3913099702390%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cabinets/material~cork&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1155K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221155K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~molykote-557-silicone-dry-film&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:37:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~molykote-food-machinery-spray-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4204073794379&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cabinets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4204073794379%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511410%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grease/trade-name~wd-40-industrial/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ultem-sheet-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cabinets/mounting-location~enclosure&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002097395%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/material~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002118662%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19013 pages (at 46 pages/min), scraped 589 items (at 0 items/min) 2025-11-02 04:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1155K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221155K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cabinets/mounting-location~enclosure/ 2025-11-02 04:38:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/threaded-rods/material~plastic/ 2025-11-02 04:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1400K2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221400K2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spot-facer-pilots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~wd-40-industrial&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ultem-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spot-weld-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002080064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:30 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:38:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spotnails-compatible-staples&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-bottles/capacity-weight-~1-1-2-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tig-torch-power-cable-adapters/torch-series-number~20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1400K2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221400K2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/cable-awg~20-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spray-bottles/capacity~64-fl-oz/capacity~1-2-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/spray-bottles/capacity-weight-~1-1-2-gal-/ 2025-11-02 04:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002080064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-can-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/union-pipe-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torch-power-cable-adapters/for-torch-series-number~20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1400K2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221400K2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=toggle-clamp-plungers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-bottles/capacity-weight-~64-fl-oz-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-nozzles/food-industry-standard~fda-compliant-21-cfr-177-2415&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tig-torch-power-cable-adapters/for-torch-series-number~20/ 2025-11-02 04:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/cable-awg~20-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002080064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/spray-bottles/capacity-weight-~64-fl-oz-/ 2025-11-02 04:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19061 pages (at 48 pages/min), scraped 589 items (at 0 items/min) 2025-11-02 04:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grinders/manufacturer-model-number~g160hd-700-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=union-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grinders/power-source~battery/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6778563655339&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spray%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6778563655339%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002111206%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:13 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3180K3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223180K3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:23 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/cable-designation~gpt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinders/manufacturer-model-number~g160hd-700-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:39:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/cable-designation~spt-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinders/power-source~battery&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-paint/color~fluorescent-green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003014881%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-paint/color~ford-gray&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7765984901538&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7765984901538%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148159%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002364450%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grinding-wheels/for-use-on~carbide/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7765984901538&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7765984901538%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076444%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3180K3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223180K3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5212847902408&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spray%20paint&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5212847902408%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002196894%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9697T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229697T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5212847902408&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=spray%20paint&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5212847902408%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002198637%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71535K22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271535K22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/industry-designation~seoow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6018T7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226018T7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7614T9&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227614T9%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7765984901538&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7765984901538%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148159%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/for-use-on~carbide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3180K3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223180K3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9697T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229697T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/material~jute&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19110 pages (at 49 pages/min), scraped 591 items (at 2 items/min) 2025-11-02 04:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7614T9&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227614T9%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/industry-designation~seoow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7765984901538&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7765984901538%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148159%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/number-of-wires~36&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512612%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:17 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9697T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229697T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trash-cans/capacity~30-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001541039%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1936T61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221936T61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8168905638803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=trash%20cans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8168905638803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107797%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7614T9&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227614T9%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5106N25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225106N25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:35 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:40:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:40:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:40:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/certification~ccc-marked/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprayers/for-use-with~water-based-paint-and-coatings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-seal-type~sealed/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grinding-wheels/grit~60/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/utility-knives/manufacturer-model-number~10-499/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9977152461181&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprayers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9977152461181%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965676%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5106N25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225106N25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprayon-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/specifications-met~ccc-marked&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/trash-cans/capacity~44-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:41:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/seal-type~sealed&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/grit~60-grit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=utility-knives/manufacturer-model-number~10-499&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cable/specifications-met~ccc-marked/ 2025-11-02 04:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19159 pages (at 49 pages/min), scraped 592 items (at 1 items/min) 2025-11-02 04:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/voltage~7-5v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5106N25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225106N25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/seal-type~sealed/ 2025-11-02 04:41:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/american-wire-gauge-awg~1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/grinding-wheels/grit~60-grit/ 2025-11-02 04:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/uv-resistant-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8540872660075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8540872660075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138837%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trash-cans/capacity-weight-~44-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tri-clamps/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/voltage~7-5v-dc-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/trash-cans/capacity-weight-~44-gal-/ 2025-11-02 04:41:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:41:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:41:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/wire-gauge~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9956631028461&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tri%20clamps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9956631028461%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trim/material~steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-barrier-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=uv-resistant-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8540872660075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8540872660075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138837%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cable/voltage~7-5v-dc-1/ 2025-11-02 04:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cable/wire-gauge~1/ 2025-11-02 04:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/material~natural-rubber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3636N19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223636N19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9948496004712&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=trim&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9948496004712%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-coils&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/hole-diameter~1-1-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/outer-circumference~38-5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/hole-diameter~2-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8540872660075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8540872660075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138837%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/hole-diameter~3-3-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/material~gum-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-loaded-drill-chuck-arbors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-fittings/tube-od~3-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-fittings/tube-od~32-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19205 pages (at 46 pages/min), scraped 593 items (at 1 items/min) 2025-11-02 04:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/v-belts/material~gum-rubber/ 2025-11-02 04:42:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~38-3125inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/for-hole-diameter~2-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:42:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~v377/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-fittings/tube-od~70-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/for-hole-diameter~3-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/v-belts/outer-circumference~38-3125inches/ 2025-11-02 04:42:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~v381/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-tube-od~3-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/grommets/for-hole-diameter~2-1-4/ 2025-11-02 04:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-tube-od~32mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/grommets/for-hole-diameter~3-3-8/ 2025-11-02 04:42:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:42:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/shape~oval/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tube-fittings/for-tube-od~3-1-8/ 2025-11-02 04:42:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tube-fittings/for-tube-od~32mm/ 2025-11-02 04:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/groove-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v377-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/length~14-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-tube-od~70mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/shape~90-elbow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v381-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tube-fittings/for-tube-od~70mm/ 2025-11-02 04:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313539%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/shape~oval&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=belt-sprockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=groove-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~18-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/length~216-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~14-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1732707636827&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1732707636827%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002308585%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/finish~phosphate-coated/color~black-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001622378%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~20-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cable-ties/length~14-ft-/ 2025-11-02 04:43:06 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/shape~oval&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19253 pages (at 48 pages/min), scraped 593 items (at 0 items/min) 2025-11-02 04:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/guides/material~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001621725%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vacuum-yor-lok-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vacuums/pickup-type~dry/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~216ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/finish~black-phosphate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bench-legs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~0-31&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cable-ties/length~216ft-/ 2025-11-02 04:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=guides/material~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/springs/finish~black-phosphate/ 2025-11-02 04:43:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/guy-line/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2891885695101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=springs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2891885695101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001598782%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~0-530&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-yor-lok-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuums/pickup-type~dry&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/guides/material~fiberglass/ 2025-11-02 04:43:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/valve-lubricants/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497672%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8306656463775&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8306656463775%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968982%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bend-and-stay-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~11-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=guy-line&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/valves/for-use-with~food/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4073N111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224073N111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valve-lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2891885695101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=springs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2891885695101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001630886%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=beveling-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hammers/head-weight~48-oz/head-weight~3-lb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hammers/head-weight~8-lb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/specifications-met~sae-as33671&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:44:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19303 pages (at 50 pages/min), scraped 594 items (at 1 items/min) 2025-11-02 04:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/for-use-with~food&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002105696%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hand-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/id~5-000-mm/id~5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hand-tap-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~4-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967679%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hammers/head-weight~3-lbs-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hammers/head-weight~8-lbs-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~5-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496892%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hammers/head-weight~3-lbs-/ 2025-11-02 04:44:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hammers/head-weight~8-lbs-/ 2025-11-02 04:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screw-mount-hand-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506618%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-tap-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/valves/valve-function~back-pressure-regulation/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50415K64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250415K64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/handles/thread-size~1-1-4-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9176T23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229176T23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tubing/id~5mm/ 2025-11-02 04:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:44:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caliper-battery-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-wheels/mount-type~screw&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bins/color~green-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/handles/thread-size~6-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handles/thread-size~1-1-4-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/valve-function~back-pressure-regulation&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7888124720991&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7888124720991%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003011043%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/handrails/component~base/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/distance-measured~0-to-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19345 pages (at 42 pages/min), scraped 596 items (at 2 items/min) 2025-11-02 04:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/od~1-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984002022255&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984002022255%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449563%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002067699%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8826536049195&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=calipers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8826536049195%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003216219%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bins/color~green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handles/thread-size~6-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000059648%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bird-screens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=92428A670&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2292428A670%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4662K46&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224662K46%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handrails/component~base&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/handsaw-saws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:45:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bins/color~green/ 2025-11-02 04:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984002022255&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984002022255%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000157155%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5840995888162&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5840995888162%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002235326%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/harsh-environment-push-button-pilot-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1743A16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221743A16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handsaw-saws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/manufacturer-model-number~798b-6-150&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/od~53-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/venturi-devices/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=call-bells&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cam-latches/key-number~2233x&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=two-hole-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:45:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8826536049195&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=calipers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8826536049195%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108035%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000113209%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2104451082401&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cam%20latches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2104451082401%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005007224%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=harsh-environment-push-button-pilot-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cams/fitting-connection~butt-weld/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19392 pages (at 47 pages/min), scraped 599 items (at 3 items/min) 2025-11-02 04:46:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-cabinets/environment~hazardous/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=venturi-devices&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hinges/hinge-type~piano/mounting-holes~without-holes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5239K19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225239K19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=11125A102&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2211125A102%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hdpe-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cams/connection-style~butt-weld&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hazardous-location-electrical-cabinets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=unfinished-steel-piano-hinges-without-holes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heat-inserts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5239K19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225239K19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=canvas&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~9-813/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cams/connection-style~butt-weld/ 2025-11-02 04:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hdpe-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cap-nuts/thread-size~3-48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cap-nuts/thread-size~m12-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heat-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7896753628184&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cap%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7896753628184%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334832%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5239K19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225239K19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heat-shrink-cord-grips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cap-nuts/thread-size~5-8-11-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~9-813/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:46:57 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cap-nuts/thread-size~3-48&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/fitting-connection~weld-socket&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cap-nuts/thread-size~m12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19436 pages (at 44 pages/min), scraped 600 items (at 1 items/min) 2025-11-02 04:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7896753628184&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cap%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7896753628184%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333424%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361151%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-shrink-cord-grips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cap-nuts/thread-size~m12/ 2025-11-02 04:47:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/handle-style~knob&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cap-nuts/thread-size~5-8-11-2&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/handle-style~pull&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9125N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229125N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/indicators/measures~heat-stress-index/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heater-temperature-controls/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969613%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968799%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heater-wire/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/pipe-size~1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4723K31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224723K31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/thread-type~bspt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000033831%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-stress-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bleeder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/video-transmission-equipment/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heater-temperature-controls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blending-discs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heater-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000140314%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:47:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:47:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helical-springs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/shaft-diameter~60-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=video-transmission-equipment&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/thread-size~1-3-4-8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blind-rivets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19486 pages (at 50 pages/min), scraped 602 items (at 2 items/min) 2025-11-02 04:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:11 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vinyl%252fbuna-n-foam-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vinyl-buna-n-foam-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helical-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-black-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/for-shaft-diameter~60mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vinyl-pipe-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/thread-size~1-3-4-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sprockets/for-shaft-diameter~60mm/ 2025-11-02 04:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/helicoils/thread-size~1-3-4-8/ 2025-11-02 04:48:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/id~196-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=viton-fluoroelastomer-rubber-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vises/drill-chuck-mount-type~integral-arbor/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/id~25-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/screw-size~5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/id~30-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blowing-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/finish~galvanized&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/id~196mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:48:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vises/mount-type~arbor&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/id~25mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550956%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/for-screw-size~0-625inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sprockets/id~196mm/ 2025-11-02 04:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/id~45-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/id~30mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7755808933160&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7755808933160%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010654%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sprockets/id~25mm/ 2025-11-02 04:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19531 pages (at 45 pages/min), scraped 602 items (at 0 items/min) 2025-11-02 04:49:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/for-screw-size~0-625inches/ 2025-11-02 04:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sprockets/id~30mm/ 2025-11-02 04:49:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/screw-size~2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thumb-nuts/nut-type~hex/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/screw-size~3-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carriage-bolts/thread-size~m12-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/id~45mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/voltage-marking-decals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7755808933160&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7755808933160%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010654%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/id~75-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sprockets/id~45mm/ 2025-11-02 04:49:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:49:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/for-screw-size~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/for-screw-size~3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/screw-size~m4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507787%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carriage-bolts/thread-size~m12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=voltage-marking-decals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-bond-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-head-thumb-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/for-screw-size~2/ 2025-11-02 04:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7755808933160&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7755808933160%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010654%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/for-screw-size~3-8/ 2025-11-02 04:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/carriage-bolts/thread-size~m12/ 2025-11-02 04:49:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:49:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-low-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/for-screw-size~m4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-pressure-sodium-lights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507787%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-bond-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=case-hardened-splines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:06 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:50:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:50:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/for-screw-size~m4/ 2025-11-02 04:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19574 pages (at 43 pages/min), scraped 602 items (at 0 items/min) 2025-11-02 04:50:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolt-bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caster-mounting-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/low-profile-side-mount-casters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-low-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-pressure-sodium-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507787%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/screw-size~5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/screw-size~9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=casters/edge-mount-bracket-casters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wall-sleeves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453979%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:38 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454380%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cat-v40-end-mill-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1981438481801&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=casters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1981438481801%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%22495168974068%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/number-of-teeth~70&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/number-of-teeth~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9994T91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229994T91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wall-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/timers/mounting-location~wall/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001528606%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522084%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/screw-size~m5-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=category-3-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wear-strips/environment~food-and-beverage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19624 pages (at 50 pages/min), scraped 603 items (at 1 items/min) 2025-11-02 04:51:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hinges/mounting-location~edge/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wd-40-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wall-timers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/relays/environment~washdown/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:19 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/screw-size~m5-5&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wear-strips/performance~food-safe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weld-nuts/thread-size~3-8-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/roller-chain-trade-number~100/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ceiling-leak-diverters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=735174209461&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wear%20strips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B735174209461%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002077504%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washdown-relays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5106733238115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=weld%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5106733238115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334816%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ceiling-tile-grid-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hinges/overall-width~4-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ceiling-tiles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheels/axle-diameter~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/roller-chain-trade-size~100-ss&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/roller-chain-trade-number~80h/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9899648339233&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9899648339233%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002084056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sprockets/roller-chain-trade-size~100-ss/ 2025-11-02 04:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cement/for-use-on~galvanized-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~4-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ceiling-tile-grid-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hinges/range-of-motion~220/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheels/wheel-diameter~4-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheels/wheel-diameter~18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2960338757520&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cement&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2960338757520%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930477%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/roller-chain-trade-size~80h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19668 pages (at 44 pages/min), scraped 603 items (at 0 items/min) 2025-11-02 04:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000122696%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9899648339233&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9899648339233%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001941800%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wipes/for-use-on~aluminum-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9899648339233&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9899648339233%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001932269%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2445N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222445N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sprockets/roller-chain-trade-size~80h/ 2025-11-02 04:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1276A91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221276A91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=center-gauges/manufacturer~starrett-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/range-of-motion~220-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8946T67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228946T67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:23 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2776320989747&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wipes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2776320989747%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929899%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:52:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6760751784651&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=center%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6760751784651%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002267107%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hinges/range-of-motion~220-/ 2025-11-02 04:52:31 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wipes/for-use-on~aluminum-2&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hobby-knives/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/cable-awg~24-3-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wipes/for-use-on~rubber-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/screw-size~no-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/material~beryllium-copper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2776320989747&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wipes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2776320989747%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930459%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/cable-awg~15-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ceramic-filled-epoxies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/holders/drill-chuck-mount-type~slot-drive-straight/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hobby-knives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/cable-awg~24-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:52 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wipes/for-use-on~rubber-2&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562241%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/for-screw-size~no-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/material~beryllium-copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:00 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:53:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire/cable-awg~24-3/ 2025-11-02 04:53:00 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=holders/mount-type~pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/washers/for-screw-size~no-4/ 2025-11-02 04:53:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510712%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8097T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228097T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19717 pages (at 49 pages/min), scraped 606 items (at 3 items/min) 2025-11-02 04:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=holders/mount-type~slot-drive-straight&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/holders/mount-type~pipe/ 2025-11-02 04:53:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/shaft-diameter~65-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-location-punches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cga-023-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98410A111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298410A111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/holders/mount-type~slot-drive-straight/ 2025-11-02 04:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cga-330-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cga-346-welding-gas-regulators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sprockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002236359%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8097T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228097T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-plugs/material~plastic-rubber-blend&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2344K92&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222344K92%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms15795-902/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms15795-904/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-location-punches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cga-510-welding-gas-regulators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-punches/hole-diameter~1-5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=145222541331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B145222541331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511016%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas1149-b0516h/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8097T1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228097T1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~ms15795-902&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas1149-b0863h/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/material~titanium-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~ms15795-904&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/for-hole-diameter~1-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/chain-style~register/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/roller-chain-trade-number~10b/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-b0516h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/washers/specifications-met~ms15795-902/ 2025-11-02 04:54:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/washers/specifications-met~ms15795-904/ 2025-11-02 04:54:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hole-punches/for-hole-diameter~1-5-16/ 2025-11-02 04:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19763 pages (at 46 pages/min), scraped 608 items (at 2 items/min) 2025-11-02 04:54:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:09 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:54:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101599%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-b0863h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas1149-d1490h/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas1149-d1732j/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/roller-chain-trade-number~c2050/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/material~titanium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/roller-chain-trade-size~10b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/register-chain-not-for-lifting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:22 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-b0516h&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101616%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/chain/material~titanium/ 2025-11-02 04:54:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 04:54:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/chain/roller-chain-trade-size~10b/ 2025-11-02 04:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas1149-dn016h/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/chain/register-chain-not-for-lifting/ 2025-11-02 04:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-hollow-sections&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-d1490h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-d1732j&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-punches/hole-diameter~6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/roller-chain-trade-size~c2050-ss&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/6-chain/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flags/u-s-state-territory~washington/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101616%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002088356%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002095250%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/chain/roller-chain-trade-size~c2050-ss/ 2025-11-02 04:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-dn016h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:54 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-d1490h&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:54:54 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-d1732j&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/for-hole-diameter~6mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:54:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002103749%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washington-flags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-slides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101616%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hole-punches/for-hole-diameter~6mm/ 2025-11-02 04:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-socket-pipe-fitting-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:07 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-dn016h&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 04:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19810 pages (at 47 pages/min), scraped 608 items (at 0 items/min) 2025-11-02 04:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-and-belt-pulleys/belt-trade-size~t2-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hollow-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-and-belt-pulley-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-barriers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6843160484390&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chain%20and%20belt%20pulleys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6843160484390%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005203617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hooks/attachment-type~swivel-hook/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=squares/shape~half-round&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-boomers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3568N101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223568N101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958961006685&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=squares&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958961006685%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002308998%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hollow-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-idlers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/water-hose-hangers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~sr1136sw/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hooks/attachment-type~swivel-hook&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~sr41w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=boring-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6392083707491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6392083707491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010998%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~sr920sw/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-hose-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-links/roller-chain-trade-number~140h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sr1136sw-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sr41w-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conveyor-brushes/bristle-material~horsehair/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-idlers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-links/roller-chain-trade-number~80h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2977408974097&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chain%20links&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2977408974097%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000102194%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19857 pages (at 47 pages/min), scraped 609 items (at 1 items/min) 2025-11-02 04:56:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2977408974097&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chain%20links&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2977408974097%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000113767%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/for-use-with~sodium-hypochlorite-bleach/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sr920sw-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-links/roller-chain-trade-number~60-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stackable-pans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=horsehair-conveyor-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wax-paper-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:24 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:56:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2977408974097&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chain%20links&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2977408974097%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000113767%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/for-use-with~bleach&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/webbing-clips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/webbing-tie-down-assemblies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-248/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-links/roller-chain-trade-size~60-2ss&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-air-control-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/for-use-with~bleach/ 2025-11-02 04:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wax-paper-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=channel-spring-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/chain-links/roller-chain-trade-size~60-2ss/ 2025-11-02 04:56:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2977408974097&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chain%20links&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2977408974097%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000113767%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=webbing-tie-down-assemblies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-248inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/weights/shape~ingot-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=check-valves/fitting-connection~luer-lock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-air-control-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/weld-bungs/pipe-material~steel-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~0-248inches/ 2025-11-02 04:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=check-valves/fitting-connection~socket-connect-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:57:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19898 pages (at 41 pages/min), scraped 609 items (at 0 items/min) 2025-11-02 04:57:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1116456421562&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=check%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1116456421562%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002362024%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=check-valves/for-use-with~acetone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-38/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1116456421562&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=check%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1116456421562%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361557%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weights/shape~ingot&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2141N4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222141N4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/weld-flanges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=box-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weld-bungs/for-pipe-material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1116456421562&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=check%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1116456421562%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965604%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7766K51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227766K51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/weights/shape~ingot/ 2025-11-02 04:57:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:57:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/weld-bungs/for-pipe-material~steel/ 2025-11-02 04:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-38inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~15-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weld-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1116456421562&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=check%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1116456421562%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965604%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~0-38inches/ 2025-11-02 04:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-65/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=check-valves/for-use-with~phosphoric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-96/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=check-valves/for-use-with~soap-solutions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~1-278/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-46875inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1116456421562&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=check%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1116456421562%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965403%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1116456421562&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=check%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1116456421562%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965378%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chemical-resistant-aprons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1116456421562&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=check%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1116456421562%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965604%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/welding-glasses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-65inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:57:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~0-46875inches/ 2025-11-02 04:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7909K91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227909K91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7909K91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227909K91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-96inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~1-278inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19943 pages (at 45 pages/min), scraped 611 items (at 2 items/min) 2025-11-02 04:58:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~0-65inches/ 2025-11-02 04:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~1-75/id~1-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~0-96inches/ 2025-11-02 04:58:14 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 04:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-glasses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~1-278inches/ 2025-11-02 04:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~2-43/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chemical-resistant-aprons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~19-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~2-9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~20-7-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/welding-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~1-75inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chokers/sling-type~wire-rope&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~1-75inches/ 2025-11-02 04:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~2-43inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~2-9inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~20-4375inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~19millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9946235349486&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chokers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9946235349486%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001633687%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~60-mm-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 04:58:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~2-43inches/ 2025-11-02 04:58:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~2-9inches/ 2025-11-02 04:58:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~20-4375inches/ 2025-11-02 04:58:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/welding-tips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~19millimeters/ 2025-11-02 04:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3550T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223550T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/thread-type~unf-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/thread-type~npt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:58:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~60millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-fittings/fitting-connection~barbed-2/for-use-with~acetic-acid-white-vinegar-5-concentration/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=circle-cotters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wheels/wheel-diameter~1-7-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-thick-wall-threaded-pipe-nipples&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/id~60millimeters/ 2025-11-02 04:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 19984 pages (at 41 pages/min), scraped 614 items (at 3 items/min) 2025-11-02 04:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/thread-type~unf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/thread-type~npt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-benders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-workbench-tops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-tips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose/thread-type~unf/ 2025-11-02 04:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/connection-style~barbed/for-use-with~acetic-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wheels/wheel-diameter~140-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheels/wheel-diameter~1-7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-fittings/fitting-connection~barbed-2/gender~female-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose-fittings/connection-style~barbed/for-use-with~acetic-acid/ 2025-11-02 04:59:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wheels/wheel-diameter~2-1-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clamps/mounting-location~through-wall/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stair-tread-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9899648339233&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9899648339233%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001922034%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-handles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-benders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheels/wheel-diameter~140-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=24025T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2224025T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/connection-style~barbed/gender~female&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheels/wheel-diameter~2-1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamps/mount-type~through-wall&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9899648339233&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9899648339233%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001933346%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hose-fittings/connection-style~barbed/gender~female/ 2025-11-02 04:59:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 04:59:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9899648339233&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9899648339233%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001934042%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=class-t-fuse-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/clamps/mount-type~through-wall/ 2025-11-02 05:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=classroom-locksets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wide-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20032 pages (at 48 pages/min), scraped 615 items (at 1 items/min) 2025-11-02 05:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hot-rolled-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9960T73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229960T73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clean-room-absorbents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clean-room-doors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wipes/for-use-on~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wide-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=standard-open-area-work-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-pipe-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hydraulic-hole-punches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hot-rolled-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/windows/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cleaners/for-use-on~air-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quick-disconnect-barbed-tube-fittings/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wipes/for-use-on~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/cable-awg~16-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9998225260164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cleaners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9998225260164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929629%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/cable-awg~18-10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-hole-punches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/cable-awg~2-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wipes/for-use-on~steel/ 2025-11-02 05:00:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/cable-awg~20-34/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-quick-disconnect-barbed-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/cable-awg~16-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/cable-awg~24-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20074 pages (at 42 pages/min), scraped 616 items (at 1 items/min) 2025-11-02 05:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/impact-torx-bit-sockets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/cable-awg~18-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/cable-awg~2-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clips/clamp-type~tweezer-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/cable-awg~20-34&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562325%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/indelible-markers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stands/safety-barrier-component~post&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561817%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561411%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9975836744491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=clips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9975836744491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001946868%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562830%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7741K17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227741K17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/diameter~0-002/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9962802582025&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stands&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9962802582025%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002067580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/cable-awg~24-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K79&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K79%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6179A4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226179A4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9204K74&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229204K74%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indelible-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9197T96&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229197T96%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561673%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=impact-torx-bit-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-002&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/indicator-mounts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/india-stones/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/indicators/for-use-with~drill-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001552957%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561673%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=closet-augers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/micrometers/manufacturer~starrett-1/manufacturer-model-number~436-1xrl-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cloth/material~aluminum-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:07 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-002&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20120 pages (at 46 pages/min), scraped 622 items (at 6 items/min) 2025-11-02 05:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cloth/material~copper-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indicator-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=india-stones&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9968537056579&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cloth&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9968537056579%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indicators/for-use-with~drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9968537056579&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cloth&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9968537056579%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000083208%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561673%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=starrett-4361xrl-3-micrometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979599325048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979599325048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964522%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=starrett-depth-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/indicators/for-use-with~hydraulic-systems/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9224T45&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229224T45%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/indicators/for-use-with~refrigerant/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98728A112&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298728A112%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:02:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:39 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inline-strainers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=starrett-series-230-micrometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inserts/for-use-in~blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indicators/for-use-with~hydraulic-systems&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indicators/for-use-with~refrigerant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979599325048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979599325048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967658%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/diameter~3-15-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~masonry&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~composite-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979599325048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979599325048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967995%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inline-strainers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inserts/for-use-in~block&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=code-61-flange-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/diameter~0-055/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1400T23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221400T23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949688381693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=coatings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949688381693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929672%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20167 pages (at 47 pages/min), scraped 624 items (at 2 items/min) 2025-11-02 05:03:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949688381693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=coatings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949688381693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003433251%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/inserts/for-use-in~block/ 2025-11-02 05:03:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~3-15mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979599325048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979599325048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967995%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/diameter~3-8-2/diameter~0-375/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inserts/thread-size~3-8-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/diameter~3-15mm/ 2025-11-02 05:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/cable-designation~cmr/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pants/application~cold-protection-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-055&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coil-spring-compressors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979599325048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979599325048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967995%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cold-shrink-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inserts/thread-size~3-8-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001540326%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/industry-designation~cmr&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cold-resistant-pants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collapsible-shovels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/diameter~3-8/ 2025-11-02 05:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9893535896715&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9893535896715%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454138%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/industry-designation~cmr/ 2025-11-02 05:03:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/outer-jacket-color~white/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collection-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/certification~rcm-marked/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulating-flat-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20214 pages (at 47 pages/min), scraped 625 items (at 1 items/min) 2025-11-02 05:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/collet-size~1-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:11 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/collet-size~15-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/collet-size~20-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/outer-insulation-color~white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3966958597415&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=collets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3966958597415%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003347850%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-chain-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/american-wire-gauge-awg~17/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/collet-size~3-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/specifications-met~rcm-marked&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3966958597415&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=collets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3966958597415%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003312013%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulating-flat-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3966958597415&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=collets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3966958597415%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003311927%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/outer-insulation-color~white/ 2025-11-02 05:04:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3223A406&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223223A406%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3966958597415&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=collets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3966958597415%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002271196%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/specifications-met~rcm-marked/ 2025-11-02 05:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3257A54&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223257A54%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3257A59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223257A59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/american-wire-gauge-awg~18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~1-1-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/wire-gauge~17&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3223A401&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223223A401%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation/for-use-on~porcelain/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-hangers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/wire-gauge~17/ 2025-11-02 05:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-coils/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3257A59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223257A59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/wire-gauge~18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~47-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/size~1-1-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/wire-brushes/shank-style~threaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-connectors/electrical-connection~straight-blade/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/for-use-on~porcelain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/wire-gauge~18/ 2025-11-02 05:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=browse-pipe-tubing%2C-hose-and-fittings/routing-and-suspending&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/collets/size~1-1-64/ 2025-11-02 05:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20259 pages (at 45 pages/min), scraped 628 items (at 3 items/min) 2025-11-02 05:05:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929673%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-coils&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation-heat-shields/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3257A59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223257A59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/browse-pipe-tubing%2C-hose-and-fittings/routing-and-suspending/ 2025-11-02 05:05:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-connectors/connection-type~straight-blade&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7479A4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227479A4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-crimping-pliers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/iron-steel-asme-pipe-flanges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/wire-brushes/shank-style~threaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=column-guards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire-connectors/connection-type~straight-blade/ 2025-11-02 05:05:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation-heat-shields&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:30 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:05:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-identification-tags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~2-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-rope-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-crimping-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~2-3-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iron-and-steel-asme-code-pipe-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/drive-size~no-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/j-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jackets/color~amber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3236610467807&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=combination%20wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3236610467807%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003488419%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~2-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-rope-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-identification-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-rope-pulleys/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~2-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=j-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/combination-wrenches/size~2-1-8/ 2025-11-02 05:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/combination-wrenches/size~2-3-8/ 2025-11-02 05:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20305 pages (at 46 pages/min), scraped 629 items (at 1 items/min) 2025-11-02 05:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-storage-drums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jackets/color~amber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~no-26/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jam-nuts/nut-type~round/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-rope-pulleys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9953543717325&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9953543717325%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548139%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-switch-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jam-nuts/thread-size~m2-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulb-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9906T101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229906T101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~no-26&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/nut-type~round&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=steel-tube-connector-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:06:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/combination-wrenches/size~no-26/ 2025-11-02 05:06:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bulkheads/fitting-connection~threaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=940728682702&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B940728682702%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000078716%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~m2-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-hangers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-saw-blades/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compound-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=940728682702&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B940728682702%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compressed-air-flow-boosters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulkheads/connection-style~threaded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-screws/length~1-13-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=step-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:03 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~m2-5&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-saw-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bulkheads/connection-style~threaded/ 2025-11-02 05:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20354 pages (at 49 pages/min), scraped 630 items (at 1 items/min) 2025-11-02 05:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jaw-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jic-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-screws/length~1-13-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compression-tube-supports&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jig-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=computer-discs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2724637663046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2724637663046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001629691%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jaw-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stepper-motors/enclosure-rating~ip65&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stepper-motors/motor-frame-size~nema-34&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jic-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=concrete-form-stakes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/joints/fitting-type~saddle-tap/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=94054A222&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2294054A222%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jig-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5513810907834&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stepper%20motors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5513810907834%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000151806%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5513810907834&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stepper%20motors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5513810907834%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001573704%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:55 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stepper-motors/enclosure-rating~ip65&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:07:56 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stepper-motors/motor-frame-size~nema-34&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:07:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sterile-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=joints/type~saddle-tap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/junction-boxes/environment~submersible/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/work-positioning-arms/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=concrete-rakes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20399 pages (at 45 pages/min), scraped 631 items (at 1 items/min) 2025-11-02 05:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=condensate-drain-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=condensate-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/joints/type~saddle-tap/ 2025-11-02 05:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stick-electrodes/aws-material-code~e7014&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conductive-lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/computer-workstations/adjustability~adjustable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=junction-boxes/environment~submersible&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=work-positioning-arms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9938597167718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stick%20electrodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9938597167718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323738%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keyed-alike-padlock-keys/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conductivity-controllers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5861588236444&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5861588236444%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000092646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit/thread-size~m50&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stock/material~noryl-ppo&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=workbenches/adjustable-height-computer-workstations-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5860541886895&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conduit&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5860541886895%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451746%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keyed-alike-padlock-keys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995947086760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=stock&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995947086760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001530052%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stones/color~green-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stock-pushers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~0910/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8316204383934&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8316204383934%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%227087751405112%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6963T99&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226963T99%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit-grommets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit-support-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8561K419&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228561K419%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8583T42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228583T42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~1041c/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stones/color~green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20451 pages (at 52 pages/min), scraped 632 items (at 1 items/min) 2025-11-02 05:09:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~0910&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bungs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/stones/color~green/ 2025-11-02 05:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400982%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=burs/shape~ball&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8583T42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228583T42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~sludge&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~1041c&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~trifluoroethane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/keys/key-number~10g050/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/certification~mil-spec/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3908531452254&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=burs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3908531452254%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002308410%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969007%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/key-number~1041c/ 2025-11-02 05:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/nema-type~l7-30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969213%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8583T42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228583T42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5428K14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225428K14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/military-specifications-met~military-specification&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=79215A328&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2279215A328%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/nema-style~l7-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~2millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/shape~15-elbow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/keys/key-number~10g050/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/connectors/military-specifications-met~military-specification/ 2025-11-02 05:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=storage-trunks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:09:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:09:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:09:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/connectors/nema-style~l7-30/ 2025-11-02 05:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~2953/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:10:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wrenches/size~2millimeters/ 2025-11-02 05:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~3581/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313540%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20497 pages (at 46 pages/min), scraped 635 items (at 3 items/min) 2025-11-02 05:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~997x/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strain-gauge-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~15-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~2953&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:10:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strainers/fitting-connection~quick-clamp-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~3581&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strainers/fitting-connection~socket-connect-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313540%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136812%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400740%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9758266108659&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=strainers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9758266108659%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361293%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9758266108659&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=strainers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9758266108659%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361557%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~a1001eh/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~205-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:10:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wye-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~a107/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wyes/fitting-connection~threaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313540%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wyes/pipe-size~3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/strainers/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081543%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~a1001eh&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~44-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wye-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~a107&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3344N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223344N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:06 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20543 pages (at 46 pages/min), scraped 635 items (at 0 items/min) 2025-11-02 05:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wyes/pipe-size~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/key-number~a1001eh/ 2025-11-02 05:11:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strainers/material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002112329%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400372%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~by15/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984228142725&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984228142725%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009285%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/strainers/material~steel/ 2025-11-02 05:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~cat5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strap-removal-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:24 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~44-ml&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:11:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3344N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223344N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/y-style-check-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~0-391-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~by15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~cat5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity~775-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~11-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3344N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223344N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454885%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401695%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=straps/component~harness&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400361%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=straps/buckle-type~spring-loaded-clamp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958071509783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=containers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958071509783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002139804%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=y-style-check-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9733419879211&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=straps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9733419879211%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002277496%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:11:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9733419879211&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=straps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9733419879211%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562117%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3344N4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223344N4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~11-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:02 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:12:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/straps/eye-type~eyeless/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~10-qt/capacity~2-5-gal/capacity~2-1-2-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20590 pages (at 47 pages/min), scraped 635 items (at 0 items/min) 2025-11-02 05:12:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~25-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~11-gal-/ 2025-11-02 05:12:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-tie-anchors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~3775-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~s26/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3344N4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223344N4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-tie-saddles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-ties/length~1-1-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=straps/eye-style~eyeless&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~2-1-2-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~25-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-tie-anchors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~3775-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/straps/eye-style~eyeless/ 2025-11-02 05:12:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~2-1-2-gal-/ 2025-11-02 05:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~s26&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~25-gal-/ 2025-11-02 05:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3344N4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223344N4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-tie-saddles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~3775-ml/ 2025-11-02 05:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~sb757/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~1-1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/key-number~s26/ 2025-11-02 05:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-ties/overall-length~10-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~sh6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~70-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001627570%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~x2511/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:12:57 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:12:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number~x2904/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~sb757&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:02 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~1-1-16&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strip-metal-benders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~10-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~sh6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20634 pages (at 44 pages/min), scraped 635 items (at 0 items/min) 2025-11-02 05:13:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~9-5-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~70-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400673%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~x2511&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001622544%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/key-number~sh6/ 2025-11-02 05:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~70-ml/ 2025-11-02 05:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~x2904&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=control-keys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400771%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3050N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223050N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zip-ties/length~8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/width~1-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~9-5-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400804%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~9-5-ml/ 2025-11-02 05:13:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-belt-drive-sprockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/width~40-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/width~1-5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/width~5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/width~1-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400804%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/material~steel/ 2025-11-02 05:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001615917%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000091340%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-brushes/flammability-rating~ul-94-v-0&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/width~40-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/width~1-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/knives/blade-type~retracting-breakaway/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/width~5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20680 pages (at 46 pages/min), scraped 636 items (at 1 items/min) 2025-11-02 05:14:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400804%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7377080483436&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20brushes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7377080483436%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116718%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000107666%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000030990%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/keys/width~5mm/ 2025-11-02 05:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-guide-mounting-rods/length~15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5891N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225891N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:21 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/width~40-mm&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knives/blade-type~retracting-breakaway&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-guide-mounting-rods/length~7-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7984217990893&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20guide%20mounting%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7984217990893%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001627802%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strut-channel-stair-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/knobs/handle-style~lever/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2846472700070&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2846472700070%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003034266%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7984217990893&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20guide%20mounting%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7984217990893%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001621748%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-guides/length~2-ft/length~24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-lane-dividers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-roller-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=39555A68&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2239555A68%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9268K562&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229268K562%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7984217990893&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20guide%20mounting%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7984217990893%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001627802%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4434376177171&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20guides&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4434376177171%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001613476%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001629723%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-rollers/between-frame-width~10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knobs/handle-style~lever&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:14:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:14:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-rollers/between-frame-width~19&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9268K402&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229268K402%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1360744530601&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1360744530601%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000054082%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/laboratory-bottles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8323758595347&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8323758595347%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969771%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20731 pages (at 51 pages/min), scraped 639 items (at 3 items/min) 2025-11-02 05:15:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7984217990893&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20guide%20mounting%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7984217990893%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001627802%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1360744530601&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1360744530601%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002267732%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:14 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knobs/handle-style~lever&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sunroc-water-cooler-replacement-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sulfuric-acid-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=super-glue-nozzles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9268K402&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229268K402%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laboratory-bottles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:27 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corbin-russwin-door-locksets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/latex-fabric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cord-grips/knockout-trade-size~m25/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-series~aquavantage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grip-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-series~royal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9268K402&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229268K402%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grip-installation-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grips/material~fiberglass-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=latex-fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grips/for-knockout-trade-size~m25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/for-manufacturer-series~aquavantage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~2570-000-001/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3111105495877&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cord%20grips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3111105495877%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003423545%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/for-manufacturer-series~royal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:15:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~269-1339/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-splitters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cord-grips/for-knockout-trade-size~m25/ 2025-11-02 05:16:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lavatory-equipment/for-manufacturer-series~aquavantage/ 2025-11-02 05:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lavatory-equipment/for-manufacturer-series~royal/ 2025-11-02 05:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20781 pages (at 50 pages/min), scraped 640 items (at 1 items/min) 2025-11-02 05:16:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~2570-000-001&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/electrical-connection~dvi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~ress-c/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~269-1339&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/voltage~1500v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swinging-bar-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123752%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swinging-latches/mounting-location~surface&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corner-shipping-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455557%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/voltage~4-5v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081745%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553669%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7693864889603&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swinging%20latches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7693864889603%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002236973%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4974T68&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224974T68%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553720%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~ress-c&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~rp3614-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~s09-007s/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7777T28&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227777T28%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lead-free-solder/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002111934%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corrosion-resistant-rollers/shoulder-adjustability~adjustable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~rp3614&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=counter-pens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~s09-007s&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553720%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20830 pages (at 49 pages/min), scraped 642 items (at 2 items/min) 2025-11-02 05:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/voltage~30v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1057451984853&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=corrosion%20resistant%20rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1057451984853%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000069141%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=counterbores/screw-size~m14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lavatory-equipment/manufacturer-model-number~rp3614/ 2025-11-02 05:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-free-solder&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553726%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/mounting-location~wall&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/leak-detecting-testers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/american-wire-gauge-awg~16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8876496971536&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=counterbores&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8876496971536%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455299%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553720%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1057451984853&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=corrosion%20resistant%20rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1057451984853%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000069141%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9241362370870&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9241362370870%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002266240%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:32 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=counterbores/screw-size~m14&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/material~leather-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=leak-detecting-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/wire-gauge~16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=counters/ul-file-number~e128604&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=counters/ul-file-number~e137808&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:45 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1057451984853&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=corrosion%20resistant%20rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1057451984853%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000069141%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cable/wire-gauge~16/ 2025-11-02 05:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/countersink-angle~120&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5519429302621&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=counters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5519429302621%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005596188%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/leds/bulb-shape-number~a15/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5519429302621&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=counters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5519429302621%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005545487%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=leather-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=switches/panel-cutout-diameter~14-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=983458193638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=countersinks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B983458193638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003441798%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4126K38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224126K38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4126K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224126K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20881 pages (at 51 pages/min), scraped 642 items (at 0 items/min) 2025-11-02 05:18:08 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coupling-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9241362370870&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9241362370870%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002214191%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:10 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/countersink-angle~120&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-chain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=LEDs/bulb-shape-number~a15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/shaft-diameter~1-9-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/left-hand-threaded-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6915K24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226915K24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9948839639468&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9948839639468%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002055044%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1796N14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221796N14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-shaft-diameter~1-9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/leds/light-technology~halogen/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~pharmaceuticals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=left-hand-threaded-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-holes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6915K24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226915K24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~sulfur-hexafluoride&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:42 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:18:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/couplings/for-shaft-diameter~1-9-16/ 2025-11-02 05:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968243%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/levels/thread-size~m16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/levers/handle-type~fixed/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969045%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4759K211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224759K211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=LEDs/light-technology~halogen&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6915K24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226915K24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~1-20-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~1-72&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9948839639468&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9948839639468%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20931 pages (at 50 pages/min), scraped 646 items (at 4 items/min) 2025-11-02 05:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~2-1-2-4-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levels/thread-size~m16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lifting-clevis-hooks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levers/handle-knob-type~stationary&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lifting-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455045%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333410%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:17 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453144%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=14685K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2214685K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4945834745524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4945834745524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000073588%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/levers/handle-knob-type~stationary/ 2025-11-02 05:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7976A22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227976A22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:26 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~1-72&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lifting-s-hooks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=93023A327&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2293023A327%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lifts/fitting-type~link-ring/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lifting-clevis-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lifting-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/squares/material~cpvc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-base-type~bayonet/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=14685K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2214685K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7976A22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227976A22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lifting-s-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-shape-number~bt56/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lifts/type~link-ring&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-shape-number~t10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cpvc-squares&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/base-type~bayonet&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=14685K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2214685K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lifts/type~link-ring/ 2025-11-02 05:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crimpers/for-wire-cable-type~power&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7976A22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227976A22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/base-type~bayonet/ 2025-11-02 05:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 20976 pages (at 45 pages/min), scraped 647 items (at 1 items/min) 2025-11-02 05:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-shape-number~t3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-shape-number~bt56&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8050600728590&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=crimpers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8050600728590%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969396%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-shape-number~t10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1016/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/length~30-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/thread-size~2-1-2-4-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:19 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:20:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002056074%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002067022%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:20:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swivels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453144%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-shape-number~t3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8352K26&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228352K26%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1016&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1611K191&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221611K191%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~30-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002057813%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/drive-size~1-3-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swivels/thread-type~anpt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~1016/ 2025-11-02 05:20:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cable-ties/length~30-ft-/ 2025-11-02 05:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=syringe-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swivels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453144%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8620K54&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228620K54%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5933689897561&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=crowfeet&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5933689897561%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489862%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swivels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002102042%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~161/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5482K874&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225482K874%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crucible-tongs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21023 pages (at 47 pages/min), scraped 650 items (at 3 items/min) 2025-11-02 05:21:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ctfe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995889107271&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=swivels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995889107271%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453144%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/size~8-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~161&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caddies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cup-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/crowfeet/size~8-mm/ 2025-11-02 05:21:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~24mb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=current-detectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~161/ 2025-11-02 05:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:21:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~2721/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=curtain-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~2723/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cushions/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~28mb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-handles/drive-size~t27/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~24mb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=curtain-walls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~2721&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-handles/drive-size~ip15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~24mb/ 2025-11-02 05:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cushions/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~2723&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~28mb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:21:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~2721/ 2025-11-02 05:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cut-resistant-chains&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-handles/drive-size~t27-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~3497/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9944536682256&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=t%20handles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9944536682256%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489874%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cushions/material~aluminum/ 2025-11-02 05:22:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~2723/ 2025-11-02 05:22:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~28mb/ 2025-11-02 05:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21071 pages (at 48 pages/min), scraped 651 items (at 1 items/min) 2025-11-02 05:22:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/t-handles/drive-size~t27-1/ 2025-11-02 05:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-handles/drive-size~ip10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/calipers/environment~washdown/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~458/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~3497&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~558/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cutters/for-cutting~din-rails-with-bracket/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutoff-wheels/arbor-hole-size~1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutters/cut-style~rounded-edge-square&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~3497/ 2025-11-02 05:22:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-handles/size~ip10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~40/ 2025-11-02 05:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/environment~washdown-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~458&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=765203497970&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cutoff%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B765203497970%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511707%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9988271414601&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cutters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9988271414601%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002234482%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/t-handles/size~ip10/ 2025-11-02 05:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/calipers/environment~washdown-1/ 2025-11-02 05:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~783/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:22:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~458/ 2025-11-02 05:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4550A203&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224550A203%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~788/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutters/for-cutting~din-rail-with-bracket&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slot-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-slots/t-slotted-framing-component~locking-collar/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~904/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cutters/for-cutting~din-rail-with-bracket/ 2025-11-02 05:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~783&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cx96-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21116 pages (at 45 pages/min), scraped 652 items (at 1 items/min) 2025-11-02 05:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~788&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~783/ 2025-11-02 05:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~efp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slots/t-slot-framing-component~locking-collar&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~904&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylinder-dollies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~788/ 2025-11-02 05:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=d-hole-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=d-ring-extenders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/t-slots/t-slot-framing-component~locking-collar/ 2025-11-02 05:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~904/ 2025-11-02 05:23:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~evc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~efp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cam-latches/finish~nickel-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~etj&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~fp39-ho/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~efp/ 2025-11-02 05:23:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2104451082401&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cam%20latches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2104451082401%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001519486%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~etj/ 2025-11-02 05:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slotted-framing-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~ftd/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dc-motor-replacement-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~evc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cam-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/density-meters/measurement-unit~specific-gravity/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:23:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~evc/ 2025-11-02 05:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~fp39-ho&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2104451082401&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cam%20latches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2104451082401%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001519486%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~ftd&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/debris-collection-tarps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~fp39-ho/ 2025-11-02 05:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21164 pages (at 48 pages/min), scraped 652 items (at 0 items/min) 2025-11-02 05:24:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=density-meters/scale~specific-gravity&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=density-meters/specifications-met~astm-e2995&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=density-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/trade-number~ftd/ 2025-11-02 05:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cap-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~0-676-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=table-leg-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2104451082401&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cam%20latches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2104451082401%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001519486%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/density-meters/scale~specific-gravity/ 2025-11-02 05:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4997713287608&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=density%20meters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4997713287608%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005650508%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/caps/pipe-size~12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~2-2-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4099K712&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224099K712%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/debris-collection-tarps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=development-board-sensor-modules/sensor-type~infrared&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~21-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~0-676w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:39 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/impact-drivers/manufacturer~dewalt-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9580658925258&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=development%20board%20sensor%20modules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9580658925258%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005389521%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~0-676w/ 2025-11-02 05:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caps/for-pipe-size~12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/25-watt-light-bulbs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~2-2w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~34-4-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~21w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/caps/for-pipe-size~12/ 2025-11-02 05:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~2-2w/ 2025-11-02 05:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9580658925258&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=development%20board%20sensor%20modules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9580658925258%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005389521%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dewalt-impact-drivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~21w/ 2025-11-02 05:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tangle-resistant-chain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~25w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21208 pages (at 44 pages/min), scraped 653 items (at 1 items/min) 2025-11-02 05:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~34-4w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:11 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~513-412-10e&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~25w/ 2025-11-02 05:25:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/light-bulbs/wattage~34-4w/ 2025-11-02 05:25:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:25:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9580658925258&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=development%20board%20sensor%20modules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9580658925258%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005389521%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dial%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002151313%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbide-insert-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/light-bulbs/wattage~8-8w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/liners/material~rubber-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~f2730iq&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~pt07136&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity~185-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/capacity~400-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:34 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dial%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/liquid-level-indicators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dial%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002093022%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002102373%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=liners/material~rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994038592548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tanks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994038592548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101571%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:43 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~f2730iq&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/liquid-level-sights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/light-bulbs/wattage~8-8w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-grinders/for-use-with~polishing-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=37415K55&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2237415K55%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/liners/material~rubber/ 2025-11-02 05:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4439T42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224439T42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=liquid-level-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lockers/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:25:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8549892642718&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=die%20grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8549892642718%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964605%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/eye-wash-solutions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=liquid-level-sights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~1-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:03 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-grinders/for-use-with~polishing-wheels&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/for-use-with~lubricant/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21258 pages (at 50 pages/min), scraped 655 items (at 2 items/min) 2025-11-02 05:26:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:26:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453222%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:26:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locks/lock-type~squeeze-to-release/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lockers/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tanks/material~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cart-bumpers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/eye-wash-solutions-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-locks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lockers/material~aluminum/ 2025-11-02 05:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/for-use-with~lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tanks/eye-wash-solutions-4/ 2025-11-02 05:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-087/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locks/lock-type~squeeze-to-release&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locks/opens-with~screwdriver&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tanks/for-use-with~lubricants/ 2025-11-02 05:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tanks/material~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locks/opens-with~combination&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971224262213&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971224262213%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001973656%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tap-aligners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971224262213&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971224262213%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968863%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tanks/material~fiberglass/ 2025-11-02 05:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~7-8-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-087&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~5-8-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971224262213&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971224262213%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968506%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~7-8-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454886%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453558%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-352/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7138A14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227138A14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449363%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:27:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caster-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21308 pages (at 50 pages/min), scraped 655 items (at 0 items/min) 2025-11-02 05:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~m2-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tape-measures/length~72/length~6-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~m30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/length~200-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453558%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-352&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~din-435/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7138A14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227138A14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000163512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334725%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5504631511088&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tape%20measures&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5504631511088%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001622923%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/length~6-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=din-6334-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-gasket-makers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453558%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=din-435-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7138A14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227138A14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tape-measures/length~6-ft-/ 2025-11-02 05:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=casting-compounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/manufacturer-model-number~33-725&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=catch-cans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:56 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:27:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/manufacturer-model-number~l1030c-02&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5504631511088&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tape%20measures&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5504631511088%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002130268%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caution-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/low-profile-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21358 pages (at 50 pages/min), scraped 655 items (at 0 items/min) 2025-11-02 05:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5504631511088&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tape%20measures&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5504631511088%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004931451%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:12 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tape-measures/manufacturer-model-number~33-725&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:28:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dirt-traps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/additive~chlorine/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=disinfectant-spray/manufacturer~lysol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=display-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=displayport-bulkheads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=low-profile-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6617809020615&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=disinfectant%20spray&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6617809020615%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002230768%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/food-industry-grease/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tapping-fluids/manufacturer~tap-magic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/additive~chlorine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7910737896220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tapping%20fluids&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7910737896220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002250048%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubriplate-no-630-aa-grease/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dmx-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001528364%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~1-1-16-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/food-industry-grease-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~1-1-4-8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dock-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:28:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dock-ramps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452916%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%222031294875466%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334635%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/heated-drum-dollies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubriplate-no-630-aa-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001528364%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21409 pages (at 51 pages/min), scraped 655 items (at 0 items/min) 2025-11-02 05:29:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4328T255&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224328T255%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~1-4-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-levers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dollies/heated-dollies-for-drums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-bells&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451929%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001528364%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~11-16-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4328T255&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224328T255%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~3-4-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/dollies/heated-dollies-for-drums/ 2025-11-02 05:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~3-56&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-levers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451148%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448449%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333442%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:46 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4328T255&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224328T255%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-peepholes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-sills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-hooks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~5-44&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:29:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-signs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnets/color~off-white-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333505%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:06 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21460 pages (at 51 pages/min), scraped 655 items (at 0 items/min) 2025-11-02 05:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~7-16-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~7-8-27&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnifying-glasses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454867%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/chain-grade~100&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnets/color~off-white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/thread-size~m20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flags/u-s-state-territory~maine/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8304400284430&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chain&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8304400284430%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002058942%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/magnets/color~off-white/ 2025-11-02 05:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/manifolds/fitting-connection~socket-connect-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnifying-glasses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/manifolds/for-use-with~helium/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:30:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/manila-rope/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-020/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=maine-flags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manifolds/connection-style~socket-connect&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manifolds/for-use-with~helium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-060/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:30:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-054&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manila-rope&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=taps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-02&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/manifolds/connection-style~socket-connect/ 2025-11-02 05:31:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=teach-pendant-mounts/for-robot-arm-manufacturer-model-number~m20ia&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8489049994451&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8489049994451%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964764%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001540539%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21508 pages (at 48 pages/min), scraped 655 items (at 0 items/min) 2025-11-02 05:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001519361%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/markers/mounting-location~in-ground/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9986406040625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=teach%20pendant%20mounts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9986406040625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110748%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-and-belt-pulleys/belt-trade-size~3vx&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-054&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-06&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:31:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6843160484390&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chain%20and%20belt%20pulleys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6843160484390%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002016079%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flags/u-s-state-territory~maryland/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8489049994451&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8489049994451%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964764%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001535587%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=markers/mount-type~in-ground&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:32 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-and-belt-pulleys/belt-trade-size~3vx&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-128&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/markers/mount-type~in-ground/ 2025-11-02 05:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-165&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-183&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001518725%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-1888&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=maryland-flags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8489049994451&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8489049994451%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964764%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/material-management-labels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-201&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000021845%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001516697%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:52 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-128&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001553077%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:31:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000143235%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-233&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:00 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-165&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:32:01 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-183&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mats/for-use-with~water/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A693&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A693%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=material-management-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:07 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-201&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21560 pages (at 52 pages/min), scraped 655 items (at 0 items/min) 2025-11-02 05:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001546024%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-239&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mats/form~pillow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flat-head-made-to-order-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:17 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-233&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:32:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=teflon-powder&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mats/for-use-with~water&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001552531%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=draft-hoods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mats/form~pillow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1629427390652&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1629427390652%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967559%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/flat-head-made-to-order-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temperature-controllers/environment~corrosive&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drains/for-use-with~sulfuric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:34 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-239&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drafting-templates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drawing-storage-cabinets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1629427390652&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1629427390652%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136682%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dowel%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%225131595517480%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9956389503914&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=temperature%20controllers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9956389503914%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055772%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9968054608189&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drains&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9968054608189%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=29595T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2229595T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5100N101&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225100N101%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7981K81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227981K81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1621T19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221621T19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temperature-sticks/temperature-points~932-f&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dressing-stones&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temperature-switches/environment~hazardous&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8273079560253&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=temperature%20sticks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8273079560253%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001979533%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temperature-switches/ul-file-number~e184396&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21614 pages (at 54 pages/min), scraped 660 items (at 5 items/min) 2025-11-02 05:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~0-38-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~11/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=830330883100&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=temperature%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B830330883100%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000137394%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3261K123&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223261K123%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=830330883100&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=temperature%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B830330883100%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004772453%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~1-1-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003250718%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~1-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~3-85-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506704%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5032K67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225032K67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509212%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~11-ga-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~4-4-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3261K123&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223261K123%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509292%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=830330883100&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=temperature%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B830330883100%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004772453%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512211%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~4-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509414%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~46/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~46/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3261K123&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223261K123%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509292%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=830330883100&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=temperature%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B830330883100%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004772453%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:33:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=test-weights/weight~5-g&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~4-ga-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~67/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21660 pages (at 46 pages/min), scraped 661 items (at 1 items/min) 2025-11-02 05:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~46-ga-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~46-ga-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1323188756785&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=test%20weights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1323188756785%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001987482%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~5-6-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001501610%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:14 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509292%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/drill-bit-size~46-ga-/ 2025-11-02 05:34:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:34:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506606%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1777T13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221777T13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509038%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~67/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~67-ga-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/for-use-on~graphite-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~214-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/drill-bit-size~67-ga-/ 2025-11-02 05:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1777T13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221777T13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thermometers/mobility~portable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermocouple-connector-adapters/thermocouple-type~n&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~67-ga-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/for-use-on~graphite&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6691521965192&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thermocouple%20connector%20adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6691521965192%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002257551%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~214mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512262%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermostat-calibration-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/for-use-on~graphite/ 2025-11-02 05:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~222-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1777T13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221777T13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermometers/mount-type~portable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001622077%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/length~214mm/ 2025-11-02 05:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3869K215&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223869K215%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21710 pages (at 50 pages/min), scraped 661 items (at 0 items/min) 2025-11-02 05:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000104183%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/thermometers/mount-type~portable/ 2025-11-02 05:35:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~95-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-point-type~notched/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:17 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/resistors/resistor-type~thick-film/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermostats/for-system-type~oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/point-angle~145&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thick-wall-aluminum-pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2297689035772&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thermostats&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2297689035772%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969084%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thickness-gauges/manufacturer-model-number~700-118-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/point-style~notched&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001525379%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thick-film-resistors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-073&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thickness-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000006151%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=17225K57&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2217225K57%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-272&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5103956777546&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thickness%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5103956777546%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089850%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/point-style~notched/ 2025-11-02 05:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000091786%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000090122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2104A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222104A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-281&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-32-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-388&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21763 pages (at 53 pages/min), scraped 663 items (at 2 items/min) 2025-11-02 05:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-397&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-42-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~10-1-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~1-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~12-2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2104A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222104A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000151792%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000066195%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000059933%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000102540%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000036957%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000097970%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~10-1mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:31 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~1-3-8&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:36:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~12-2mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2104A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222104A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000066195%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/shank-size~10-1mm/ 2025-11-02 05:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/shank-size~12-2mm/ 2025-11-02 05:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~15-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~2-65-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-sealants/for-use-with~ammonia&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~25-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-sealants/for-use-with~liquid-ammonia&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:36:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119862%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000066195%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000081274%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7597890316846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7597890316846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965242%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000136720%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~4-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7597890316846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7597890316846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967188%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-sealants/for-use-with~r-134a-refrigerant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~4-6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21815 pages (at 52 pages/min), scraped 663 items (at 0 items/min) 2025-11-02 05:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013554%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7597890316846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7597890316846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984539%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:14 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~5-75-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4538K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224538K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thread-sealants/for-use-with~sulfuric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~4-6mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~5-7-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-broom-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/thread-size~1-1-2-6-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7597890316846&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=thread%20sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7597890316846%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/shank-size~4-6mm/ 2025-11-02 05:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~5-75mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000130636%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/thread-size~1-12-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/thread-size~1-3-8-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7151885265529&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=threaded%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7151885265529%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=45855K88&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2245855K88%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/shank-size~5-75mm/ 2025-11-02 05:37:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7151885265529&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=threaded%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7151885265529%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333464%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7151885265529&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=threaded%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7151885265529%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448969%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:37:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000130636%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A767&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A767%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~1-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/through-hole-resistors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=threaded-rods/thread-size~9-64-41-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~16-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21865 pages (at 50 pages/min), scraped 665 items (at 2 items/min) 2025-11-02 05:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6635333760770&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6635333760770%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001587524%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7151885265529&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=threaded%20rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7151885265529%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449205%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~75-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/material~nylon-fabric-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6635333760770&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6635333760770%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001605478%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000130636%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-cups&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A767&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A767%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~1-1-8&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:38:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6635333760770&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6635333760770%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001624389%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6635333760770&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6635333760770%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512483%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:21 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ties/length~16-ft&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:38:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6350K689&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226350K689%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/through-hole-resistors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9652K67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229652K67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:32 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98935A767&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298935A767%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tig-torch-power-cable-adapters/torch-series-number~22/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torch-nozzles/tig-industry-number~54n16sc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torch-gas-diffusers/for-electrode-diameter~0-094&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6350K689&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226350K689%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2215634484173&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tig%20torch%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2215634484173%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004779310%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torches/torch-series-number~9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:48 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9995116477386&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tig%20torch%20gas%20diffusers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9995116477386%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000165029%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tile-saddles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:38:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torch-power-cable-adapters/for-torch-series-number~22&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8171926610323&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tig%20torches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8171926610323%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002102679%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=timing-belts/belt-trade-number~16-at5-375&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:38:58 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tig-torch-gas-diffusers/for-electrode-diameter~0-094&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:00 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:39:00 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:39:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6350K689&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226350K689%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tig-torch-power-cable-adapters/for-torch-series-number~22/ 2025-11-02 05:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7222310939460&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=timing%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7222310939460%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005295813%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21917 pages (at 52 pages/min), scraped 666 items (at 1 items/min) 2025-11-02 05:39:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=timing-belts/belt-trade-number~25-at5-300&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tire-air-nozzles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tire-chocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5863N25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225863N25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tire-sealants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7222310939460&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=timing%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7222310939460%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005297866%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=titanium-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=toggle-clamp-replacement-spindle-components&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=toilet-and-urinal-partitions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5863N35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225863N35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=toilet-tissue-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tool-hanging-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tool-sheaths&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tool-storage-wrap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torch-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5863N35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225863N35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torlon-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torque-limiters/shaft-diameter~1-2-2/shaft-diameter~0-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torr-epoxies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4201406204775&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=torque%20limiters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4201406204775%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000162952%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000032254%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/totes/color~green-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 21969 pages (at 52 pages/min), scraped 667 items (at 1 items/min) 2025-11-02 05:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=totes/capacity~3-1-4-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/trammels/manufacturer~starrett-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5863N35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225863N35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=totes/capacity~88-fl-oz&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trailer-fence&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8786789701969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=totes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8786789701969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079859%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 503 Service Unavailable 2025-11-02 05:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:18 [scrapy.spidermiddlewares.httperror] (PID: 148) INFO: Ignoring response <503 https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5863N35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225863N35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true>: HTTP status code is not handled or not allowed 2025-11-02 05:40:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:40:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=transfer-tanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trident-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=totes/color~green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trammels/manufacturer~starrett&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trimetal-brazing-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/totes/color~green/ 2025-11-02 05:40:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8786789701969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=totes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8786789701969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079859%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/trammels/manufacturer~starrett/ 2025-11-02 05:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=trouble-lights/light-technology~incandescent&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=triserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=truck-gates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=659715627913&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=trouble%20lights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B659715627913%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968676%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-cutters-for-flexible-nonmetallic-conduit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-end-caps/tube-od~22-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8786789701969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=totes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8786789701969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079859%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-use-with~ammonium-hydroxide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:40:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8424K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228424K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-fittings/for-use-with~plastic-pellet/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-use-with~chlorine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-use-with~compressed-air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7659948933498&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20end%20caps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7659948933498%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002215936%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968052%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22023 pages (at 54 pages/min), scraped 668 items (at 1 items/min) 2025-11-02 05:41:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tubing/color~off-white-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965544%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7091553161552&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7091553161552%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967611%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/color~maroon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5269K606&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225269K606%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:16 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3441N31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223441N31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-fittings/for-use-with~plastic-pellets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002196457%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tube-fittings/for-use-with~plastic-pellets/ 2025-11-02 05:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/color~off-white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/food-industry-standard~fda-compliant-21-cfr-177-1550-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9590K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229590K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/id~0-054&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/tubing/color~off-white/ 2025-11-02 05:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3441N31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223441N31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002101397%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/system-of-measurement~metric/material~stainless-steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing/manufacturer-equivalent-number~b-44-4x&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing-coilers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001492849%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tubing-reels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2980143699607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2980143699607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148043%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tungsten-carbide-sheet-metal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3441N31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223441N31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:41:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K613&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K613%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tungsten-copper-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6546T23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226546T23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22075 pages (at 52 pages/min), scraped 670 items (at 2 items/min) 2025-11-02 05:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=turnbuckle-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=turnbuckles/thread-size~1-4-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=turnbuckles/thread-size~4-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=turning-holders-for-carbide-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezers/material~polypropylene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3217042704133&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=turnbuckles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3217042704133%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3217042704133&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=turnbuckles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3217042704133%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000122836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezers/tweezer-trade-number~124&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3656929274017&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tweezers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3656929274017%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509869%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:32 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=turnbuckles/thread-size~1-4-20&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezers/tweezer-trade-number~249&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezers/tweezer-trade-number~33a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezers/tweezer-trade-number~5wf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3656929274017&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tweezers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3656929274017%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004291856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=twine/material~cotton-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3656929274017&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tweezers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3656929274017%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004248590%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3656929274017&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tweezers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3656929274017%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004248573%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3656929274017&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=tweezers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3656929274017%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004694189%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3217042704133&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=turnbuckles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3217042704133%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000122836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:47 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezers/tweezer-trade-number~124&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4722481191354&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=twine&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4722481191354%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003432833%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=two-way-flow-rotating-joints&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:55 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezers/tweezer-trade-number~33a&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:42:56 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tweezers/tweezer-trade-number~5wf&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1929T41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221929T41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=type-b-tapping-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/radios/radio-frequency-band~uhf/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22128 pages (at 53 pages/min), scraped 672 items (at 2 items/min) 2025-11-02 05:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3217042704133&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=turnbuckles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3217042704133%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000122836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=u-cup-installation-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=u-tube-manometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ultrasonic-knife-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ultraviolet-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=un-compliant-jugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=uhf-radios&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:43:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=urinal-flush-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=urethane-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=usb-charging-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=utility-knives/manufacturer-model-number~11-983&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~276-h3m-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~129-h3m-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~300j6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~320j10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~320j8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8540872660075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=utility%20knives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8540872660075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123323%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005296952%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005295727%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~3280-j8m-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283985%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~3l290&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282770%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282918%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4927A35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224927A35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7939K525&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227939K525%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005295799%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7939K515&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227939K515%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22183 pages (at 55 pages/min), scraped 673 items (at 1 items/min) 2025-11-02 05:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283283%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~3v280&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:44:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281641%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~490j10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~4l275&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~4l460&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~535-r5m-15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~5l460&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~5vx2000&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~ax33&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~880-j8m-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281137%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005284403%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282136%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005295281%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005284151%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005284305%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283630%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~b140&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005296734%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:44:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281890%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22233 pages (at 50 pages/min), scraped 676 items (at 3 items/min) 2025-11-02 05:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005295281%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005296734%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~b24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~b47&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~bb103&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~bx100&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~bx68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~bb92&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282649%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281563%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005312510%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005295281%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005284662%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282555%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~bx82&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005312660%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005296734%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281090%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:45:51 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~c71&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~c54&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~c83&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~xpa1282&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:45:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~xpz1312&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/belt-trade-number~xpz987&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283483%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283276%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282766%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005283730%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~1030-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22288 pages (at 55 pages/min), scraped 676 items (at 0 items/min) 2025-11-02 05:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~131&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005282666%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005281881%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/outer-circumference~30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000042620%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000143299%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~v344/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~40-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~670-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~875-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~v377/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-belts/outer-circumference~913-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000029523%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/outer-circumference~30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000036414%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000158694%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5134135264830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=v%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5134135264830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000134295%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v344-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:46:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valve-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/handle-type~dual-standard&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22338 pages (at 50 pages/min), scraped 676 items (at 0 items/min) 2025-11-02 05:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=valves/valve-type~dual-check&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=variable-speed-electric-mixers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=variance-indicator-contact-points/manufacturer-model-number~74-111492&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=variance-indicators/manufacturer-model-number~2424a-19cal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=variance-indicators/manufacturer-model-number~513-404-10e&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vent-hoods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001932877%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9394720567812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9394720567812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002312909%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9657484587707&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=variance%20indicator%20contact%20points&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9657484587707%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005330348%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vented-bottles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9016360397335&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=variance%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9016360397335%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002087578%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9016360397335&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=variance%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9016360397335%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2892N12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222892N12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4775K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224775K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vents/shape~rectangle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vertical-stages&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9016360397335&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=variance%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9016360397335%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002087578%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vibration-damping-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2491040633414&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=vents&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2491040633414%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002309280%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vibration-meters/measures~acceleration&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wafer-chucks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=walk-through-boot-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:47:58 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vents/shape~rectangle&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vises/manufacturer-model-number~8220&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wall-calendars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4981317823553&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=vibration%20meters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4981317823553%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002105187%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9016360397335&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=variance%20indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9016360397335%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002087578%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/warning-lights/bulb-type~halogen/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22389 pages (at 51 pages/min), scraped 678 items (at 2 items/min) 2025-11-02 05:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7755808933160&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=vises&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7755808933160%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004703625%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wall-mount-stack-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wall-patches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4729N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224729N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4981317823553&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=vibration%20meters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4981317823553%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002105187%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wash-fountain-repair-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=warning-lights/light-technology~halogen&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wash-hands-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/color~purple&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/screw-size~m150/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/screw-size~m39/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:48:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3067541930726&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=warning%20lights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3067541930726%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/finish~painted&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001534191%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4981317823553&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=vibration%20meters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4981317823553%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002105187%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~an960-1216l/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms15795-421/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539034%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/material~bumax-109-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/for-screw-size~m150&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/for-screw-size~m39&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3067541930726&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=warning%20lights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3067541930726%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510394%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001534191%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:48:59 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:49:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/for-screw-size~m150/ 2025-11-02 05:49:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/for-screw-size~m39/ 2025-11-02 05:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~an960-1216l&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22439 pages (at 50 pages/min), scraped 679 items (at 1 items/min) 2025-11-02 05:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms16212-14/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms15795-421&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:09 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/material~bumax-109-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:49:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms27183-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms27183-18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms27183-25/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~an960-1216l/ 2025-11-02 05:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3067541930726&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=warning%20lights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3067541930726%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms15795-421/ 2025-11-02 05:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001534191%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms27183-51/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms51496p67/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms51496p85/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms16212-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms27183-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms27183-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms27183-25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:34 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms16212-14/ 2025-11-02 05:49:38 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:49:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms27183-5/ 2025-11-02 05:49:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms27183-18/ 2025-11-02 05:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms27183-51&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms27183-25/ 2025-11-02 05:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms51496-p67&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/military-specifications-met~ms51496p85&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms27183-51/ 2025-11-02 05:49:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:52 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:49:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms51496-p67/ 2025-11-02 05:49:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/screw-size~m40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/washers/military-specifications-met~ms51496p85/ 2025-11-02 05:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/screw-size~m52&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~astm-d5213&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:49:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-b0763h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-d0632j&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-d0716h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448825%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453475%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22488 pages (at 49 pages/min), scraped 679 items (at 0 items/min) 2025-11-02 05:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002077699%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002113845%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002084196%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002151397%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-dn516j&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-dn832h&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/screw-size~m52&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:50:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:50:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:50:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:22 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-d0632j&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:50:22 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-d0716h&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002118430%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110318%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/profile~raised-grid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/for-plumbing-fixture~washing-machine/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-f0332p&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:33 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-dn832h&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-f0732p&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1515-h9l-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002130730%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002118430%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002106076%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107697%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas1149-f1490p/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/washer-type~raised-grid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washing-machine-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=watches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:48 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=washers/specifications-met~nas1149-f0732p&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:50:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001982034%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002118430%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:50:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-cooler-water-bottles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-leak-detectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:02 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:51:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:51:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas1149-f1490p/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-shock-absorbers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22538 pages (at 50 pages/min), scraped 679 items (at 0 items/min) 2025-11-02 05:51:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-tracers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wax-pencils&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=way-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weather-temperature-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weights/shape~ingot-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:51:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welded-chain-for-lifting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 05:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4623737981840&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=weights&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4623737981840%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310340%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-check-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-gas-flashback-arresters/for-use-with~oxygen&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-gas-flashback-arresters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9031K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229031K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-gas&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5542661873682&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=welding%20gas%20flashback%20arresters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5542661873682%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:47 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-gas-flashback-arresters/for-use-with~oxygen&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-table-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-torch-check-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-wire/aws-material-code~er70s-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002113845%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-wire/for-joining~cast-iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-wire-conduit-assemblies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7852200896140&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=welding%20wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7852200896140%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002325995%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wet-mops/mop-head-durability~disposable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002113845%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22589 pages (at 51 pages/min), scraped 680 items (at 1 items/min) 2025-11-02 05:52:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7852200896140&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=welding%20wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7852200896140%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454217%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6072169312006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wet%20mops&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6072169312006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002051135%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheel-brush-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7317T16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227317T16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheeled-stools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:19 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:52:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=winch-straps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wind-screens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002130730%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-frames&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107697%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002130730%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-channels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:37 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:52:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/welding-collets/tig-industry-number~10n23/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8607689574665&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=washers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8607689574665%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107697%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-openers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cam-lock-levers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drinking-water-filters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heavy-jam-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-185/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=10n23-welding-collets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:52:57 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/thread-size~m3-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/camera-mounts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cam-lock-levers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drinking-water-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22637 pages (at 48 pages/min), scraped 681 items (at 1 items/min) 2025-11-02 05:53:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cooling-fluids/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/campbell-hausfeld-compatible-staples/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:53:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:53:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:53:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/die-grinders/head-angle~98/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-185&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~m3-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/3d-printer-filaments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=camera-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/90-degree-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bronze-alloy-932-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498289%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000028439%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cooling-fluids&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=campbell-hausfeld-compatible-staples&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-grinders/head-angle~98-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89955K348&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289955K348%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/conductive-easy-to-print-3d-printer-filaments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/material~936-bearing-bronze/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=90-degree-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-alloy-932-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/die-grinders/head-angle~98-/ 2025-11-02 05:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-key-stock-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/3d-printer-filaments/conductive-easy-to-print-3d-printer-filaments/ 2025-11-02 05:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~a112/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-l-channels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-pullers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-alloy-936-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-size~1-4-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/copper-alloy-110-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-key-stock-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22680 pages (at 43 pages/min), scraped 682 items (at 1 items/min) 2025-11-02 05:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/certification~msha-accepted&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a112-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-l-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~ax67/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~a144/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~1-4-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002357814%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/copper-alloy-102/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper-alloy-110-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-191/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334827%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6584T17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226584T17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/150-pound-flanges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ax67-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a144-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~0-391-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper-alloy-102&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-191&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-007&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-size~1-8-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/check-valves/fitting-connection~unthreaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536272%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=150-pound-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cpvc-(schedule-80)-pipe/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001534857%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/3d-printer-cleaners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:54:57 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-191&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-205/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=check-valves/connection-style~unthreaded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22723 pages (at 43 pages/min), scraped 683 items (at 1 items/min) 2025-11-02 05:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~9-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/check-valves/for-use-with~carbon-dioxide/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cpvc-(schedule-80)-pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bolts/thread-size~1-8/ 2025-11-02 05:55:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/check-valves/connection-style~unthreaded/ 2025-11-02 05:55:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000023509%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chamfer&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/17-4-ph-stainless-steel-shim-stock-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-205&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~er17-33wkp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=check-valves/for-use-with~carbon-dioxide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~7305-2rs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001524608%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reversible-humidity-cards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=change-gears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/military-specification~ms124711/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1116456421562&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1116456421562%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964834%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=17-4-ph-stainless-steel-shim-stock-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:40 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-205&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~er17-33wkp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4851K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224851K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bronze-mesh/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~7305-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/18-8-stainless-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reversible-humidity-cards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116987%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/military-specifications-met~ms124711&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092505%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=character-punches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2286N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222286N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4851K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224851K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/helicoils/military-specifications-met~ms124711/ 2025-11-02 05:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-mesh&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=18-8-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22770 pages (at 47 pages/min), scraped 683 items (at 0 items/min) 2025-11-02 05:56:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-size~3-4-10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-4425/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~1-2-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2286N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222286N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-size~3-4-6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/compressed-air-moisture-separators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4851K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224851K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~1-2-40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~3-4-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~1-2-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334890%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cpvc-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-4425&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~3-4-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compressed-air-moisture-separators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:39 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001534857%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2286N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222286N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~1-2-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453742%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:56:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001495781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455616%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001534857%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~1-3-8-6-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cpvc-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~c63/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:56:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-508/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-plates/thickness~0-062/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22814 pages (at 44 pages/min), scraped 683 items (at 0 items/min) 2025-11-02 05:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cpvc-tube-elbows/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455616%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~1-3-8-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drum-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/45v26-welding-gas-lenses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=c63-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-508&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/dies/thread-size~1-3-8-6/ 2025-11-02 05:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~0-062&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cpvc-tube-elbows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455616%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001502322%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9495772993241&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9495772993241%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001457137%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-51/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=45v26-welding-gas-lenses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drum-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-plates/thickness~0-09/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/thread-size~1-1-4-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1651T3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221651T3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7051711979057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=helicoils&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7051711979057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/check-valves/for-use-with~xylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:57:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~0-09&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:57:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-duct-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/retractable-spring-plungers/fastener-head-type~l-handle/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/compression-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~2-753/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9495772993241&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9495772993241%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001467155%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22857 pages (at 43 pages/min), scraped 684 items (at 1 items/min) 2025-11-02 05:58:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-063&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=check-valves/for-use-with~xylene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89015K221&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289015K221%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-duct-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/l-handle-retractable-spring-plungers-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550926%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/length~made-to-order&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compression-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~2-753&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1116456421562&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1116456421562%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969097%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pins/l-handle-retractable-spring-plungers-7/ 2025-11-02 05:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313681%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070503%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/copper-square-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=46505K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2246505K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drum-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550926%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/208mxl-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4620N111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224620N111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-071&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper-square-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-plates/thickness~2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/5s-carts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001528805%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drum-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550926%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-445/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=208mxl-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:59:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:59:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~a84/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22903 pages (at 46 pages/min), scraped 687 items (at 3 items/min) 2025-11-02 05:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~kr20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5s-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:59:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-445&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9495772993241&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9495772993241%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001453258%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drums/capacity~2-1-2-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/vibration-resistant-stud-anchors-for-concrete&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001502322%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539364%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brooms/bristle-firmness~soft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a84-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9246K681&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229246K681%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~kr20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/anchors/vibration-resistant-stud-anchors-for-concrete/ 2025-11-02 05:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001502322%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:29 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-445&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 05:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drums/capacity-weight-~2-1-2-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002118548%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/thread-size~8-36&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/caps/pipe-size~3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/caps/pipe-size~6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brooms/bristle-firmness~soft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9246K681&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229246K681%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drums/capacity-weight-~2-1-2-gal-/ 2025-11-02 05:59:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 05:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7685K3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227685K3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7051711979057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=helicoils&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7051711979057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334660%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:47 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 05:59:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/thread-size~m14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9318316703734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9318316703734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001998631%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/25-pin-db-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 05:59:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caps/for-pipe-size~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7051711979057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=helicoils&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7051711979057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-9385/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caps/for-pipe-size~6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9246K681&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229246K681%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~gyae35rrb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/caps/for-pipe-size~3/ 2025-11-02 06:00:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/compression-tube-straights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22949 pages (at 46 pages/min), scraped 688 items (at 1 items/min) 2025-11-02 06:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/caps/for-pipe-size~6/ 2025-11-02 06:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=25-pin-db-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-9405/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rigid-plastic-conduit/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~1-3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-9385&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~230-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~gyae35rrb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505193%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~10-013-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002077770%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/compressor-mufflers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-9405&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rigid-plastic-conduit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/compressor-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~1-3-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~230-000mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505233%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~1-3-mm/ 2025-11-02 06:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505193%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~230-000mm/ 2025-11-02 06:00:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~r10-2z/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~10-013mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compressor-mufflers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chemical-wipes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drums/capacity~95-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=compressor-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/280-56-batteries/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~10-013mm/ 2025-11-02 06:01:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:01:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 22993 pages (at 44 pages/min), scraped 688 items (at 0 items/min) 2025-11-02 06:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~mr9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-024/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/caps/performance~tamper-evident/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~r10-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helmet-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/abrasive-grains/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drums/capacity-weight-~95-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=280-56-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chemical-wipes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002130685%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505193%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drums/capacity-weight-~95-gal-/ 2025-11-02 06:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~mr9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-024&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caps/plug-type~tamper-evident&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-rods/material~aluminum-2/material~6020-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~25-0508-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrasive-grains&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002129719%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/balls/diameter~0-09/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496104%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/caps/plug-type~tamper-evident/ 2025-11-02 06:01:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:01:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-inspection-mirrors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7604K931&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227604K931%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5560K654&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225560K654%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-rods/material~aluminum/material~6020-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~26-0508-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~25-0508mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balls/diameter~0-09&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hemp-rope&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hepa-filter-elements&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-inspection-mirrors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510688%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~25-0508mm/ 2025-11-02 06:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23038 pages (at 45 pages/min), scraped 690 items (at 2 items/min) 2025-11-02 06:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~4-0000-mm/diameter~4-000-mm/diameter~4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001546406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-type~unf-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~27-035-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9038K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229038K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cord-grips/knockout-trade-size~1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~11-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cord-grips/knockout-trade-size~2-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/marine-pins/button-style~recessed/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~26-0508mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~4mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001546406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-type~unf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~27-035mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~26-0508mm/ 2025-11-02 06:02:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/dowel-pins/diameter~4mm/ 2025-11-02 06:02:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grips/for-knockout-trade-size~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-grips/for-knockout-trade-size~2-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/removable-bolt-anchors-for-concrete-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bolts/thread-type~unf/ 2025-11-02 06:02:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~27-035mm/ 2025-11-02 06:02:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowd-control-gates/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dryer-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001546406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cord-grips/for-knockout-trade-size~1/ 2025-11-02 06:02:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cord-grips/for-knockout-trade-size~2-1-2/ 2025-11-02 06:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pins/removable-bolt-anchors-for-concrete-8/ 2025-11-02 06:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~10-4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubrication-brushes/handle-material~metal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-mesh-fencing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~28-051-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~0-164&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chicago-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowd-control-gates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dryer-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23080 pages (at 42 pages/min), scraped 691 items (at 1 items/min) 2025-11-02 06:03:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:11 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001523418%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~10-4-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-lubrication-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-mesh-fencing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~28-051mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~10-4-mm/ 2025-11-02 06:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8890K165&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228890K165%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-069/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carabiners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chicago-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~11-1-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:03:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~28-051mm/ 2025-11-02 06:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-mop-pails/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/cable-awg~18-6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~nic1117/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~b270/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8890K165&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228890K165%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-069&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carabiners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-sheets/material~aluminum-2/material~6063-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~11-1-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-mop-pails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001499111%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~11-1-mm/ 2025-11-02 06:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/cable-awg~18-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~nic1117&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:03:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=b270-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8890K165&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228890K165%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K712&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K712%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/boots/protects-against~solvents/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/material~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/material~aluminum/material~6063-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562974%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114585%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23125 pages (at 45 pages/min), scraped 691 items (at 0 items/min) 2025-11-02 06:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509824%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~7-8-6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/cable-awg~20-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=boots/protects-against~solvents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/material~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4490T111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224490T111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/dies/thread-size~7-8-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114585%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-sheets/thickness~0-01/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9986634754830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9986634754830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002344741%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/dowel-pins/material~stainless-steel/ 2025-11-02 06:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~7-8-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~7-8-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/cable-awg~20-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~9-16-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4490T111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224490T111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114585%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451921%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562372%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/thickness~0-01&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~7-8-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8219K57&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228219K57%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001465078%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~9-16-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=circuit-panel-stickers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4490T111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224490T111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acetal-homopolymer-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454626%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452902%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23174 pages (at 49 pages/min), scraped 693 items (at 2 items/min) 2025-11-02 06:05:08 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chrome-moly-pipe-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-power-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/cage-material~bronze-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dies/thread-size~m14/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/condenser-tube-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-homopolymer-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rail-fittings/material~carbon-fiber-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-sheets/thickness~0-047/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carbon-fiber-sheet-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chrome-moly-pipe-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/cage-material~bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dies/thread-size~m14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/robot-calibration-styli/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=condenser-tube-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/boss-hose-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/cage-material~bronze/ 2025-11-02 06:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-fiber-rail-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9850566353046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9850566353046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/thickness~0-047&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-fiber-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/building-wire/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/balls/diameter~23-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001466811%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/diameter~1-25-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-calibration-styli&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=boss-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23213 pages (at 39 pages/min), scraped 693 items (at 0 items/min) 2025-11-02 06:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~19-ga-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cord-keepers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=building-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~19-0-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balls/diameter~23-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cord-organizers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~30-05-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~1-25mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acoustical-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~19-ga&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-keepers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001531901%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire/diameter~1-25mm/ 2025-11-02 06:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/balls/diameter~3-937/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~19-ga/ 2025-11-02 06:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-organizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/b77-v-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~30-05mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bottles/capacity~1-8-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acoustical-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~19-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001531901%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~30-05mm/ 2025-11-02 06:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/thread-size~m5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balls/diameter~3-937&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~1-55-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~19-mm/ 2025-11-02 06:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=b77-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548403%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/b81-v-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bottles/capacity-weight-~1-8-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001545546%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23256 pages (at 43 pages/min), scraped 693 items (at 0 items/min) 2025-11-02 06:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/curtains/color~dark-green/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001531901%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acrylic-cleaners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/thread-size~m5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bottles/capacity-weight-~1-8-ml/ 2025-11-02 06:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A223&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A223%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000011539%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/access-door-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548403%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/balls/diameter~5-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=b81-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=curtains/color~dark-green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:27 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acrylic-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/curtains/color~orange/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/b90-v-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8163625184607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8163625184607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002203177%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=access-door-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548403%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct/for-use-with~wood-chips/for-use-with~wood-shavings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balls/diameter~5-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:49 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=curtains/color~dark-green&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 06:07:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drafting-equipment/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conduit/thread-size~5-8-11-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536855%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=curtains/color~orange&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=b90-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chucks/drill-chuck-mount-type~slot-drive-straight/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:02 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/for-use-with~chips-and-shavings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6490K66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226490K66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8163625184607&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8163625184607%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001526091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23299 pages (at 43 pages/min), scraped 694 items (at 1 items/min) 2025-11-02 06:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drafting-equipment&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamping-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/duct/for-use-with~chips-and-shavings/ 2025-11-02 06:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit/thread-size~5-8-11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/connects-to~electrical-box/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/balls/diameter~7-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chucks/mount-type~slot-drive-straight&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/conduit/thread-size~5-8-11/ 2025-11-02 06:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-chuck-arbors/for-machine-spindle-type~straight/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cushion-grip-screwdrivers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chucks/mount-type~slot-drive-straight/ 2025-11-02 06:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bulkheads/for-use-with~calcium-chloride/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/drive-size~t4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/connection-type~electrical-box&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/cable-awg~8-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=balls/diameter~7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2653472997099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20drivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2653472997099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003488436%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cords/connection-type~electrical-box/ 2025-11-02 06:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/thread-size~m2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chucks/straight-shank-drill-chuck-arbors-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cushion-grip-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9949660162619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9949660162619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001537651%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulkheads/for-use-with~calcium-chloride&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334645%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acetal-check-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6740743282145&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6740743282145%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%222017105433294%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~3mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/cable-awg~8-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:08:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carbon-steel-strips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9963946302057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9963946302057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2811A53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222811A53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~3mm/ 2025-11-02 06:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/robotics/robot-cell-component~vacuum-lifter/for-robot-arm-manufacturer~precise-automation/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23343 pages (at 44 pages/min), scraped 695 items (at 1 items/min) 2025-11-02 06:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561074%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acetal-copolymer/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-check-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acetal-hammers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9697T6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229697T6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-steel-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9963946302057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9963946302057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamps-for-gum-rubber-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-385/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-copolymer&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bottles/capacity~30-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/nema-type~l14-30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/standard-ball-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9963946302057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9963946302057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/digital-weight-scales/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/voltage~6v-dc-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-385&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bottles/capacity-weight-~30-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/press-fit-drill-bushings-with-head/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/ball-bearings-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507898%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:09:56 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:09:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bottles/capacity-weight-~30-ml/ 2025-11-02 06:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/cable-designation~seow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=digital-weight-scales&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/ball-bearings-8/ 2025-11-02 06:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/voltage~6v&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23387 pages (at 44 pages/min), scraped 697 items (at 2 items/min) 2025-11-02 06:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~0-575/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/clamping-and-workholding/press-fit-drill-bushings-with-head-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-storage-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/batteries/voltage~6v/ 2025-11-02 06:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507898%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~216-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pip-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/voltage~12v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/industry-designation~seow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acetal-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cutoff-inserts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-575&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cable/industry-designation~seow/ 2025-11-02 06:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-storage-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507898%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/battery-chargers/battery-voltage~20v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~216-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001500409%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/voltage~12v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~46-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003931320%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutoff-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8950K681&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228950K681%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bulkheads/thread-type~nptf/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:52 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:10:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-chargers/for-battery-voltage~20v&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-strut-mount-pipe-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70235K14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270235K14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carriage-and-plow-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~46mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/battery-chargers/for-battery-voltage~20v/ 2025-11-02 06:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clamp-on-lights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-sump-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23429 pages (at 42 pages/min), scraped 698 items (at 1 items/min) 2025-11-02 06:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003931320%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bulkheads/thread-type~nptf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/din-6796-lock-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~46mm/ 2025-11-02 06:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adjustable-height-chairs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-strut-mount-pipe-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9963946302057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9963946302057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carriage-and-plow-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamp-on-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-sump-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~1-63-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003931320%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50785K736&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250785K736%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=din-6796-lock-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-height-chairs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carriages/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~1-63-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:47 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bullet-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~3-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/cable-designation~thwn/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cone-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013838%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tie-down-anchors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carriages&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~1-35-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550245%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bullet-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23474 pages (at 45 pages/min), scraped 700 items (at 2 items/min) 2025-11-02 06:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=88605K49&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2288605K49%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/industry-designation~thwn&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cone-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013838%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/voltage~28000v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cable/industry-designation~thwn/ 2025-11-02 06:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/roll-formers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~1-35mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acetylene-labels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duplex-receptacles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~1-2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bumper-guards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~3-7-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013838%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/diameter~1-35mm/ 2025-11-02 06:12:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/voltage~28000v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550976%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roll-formers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553819%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetylene-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duplex-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~1-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bumper-guards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A272&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A272%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:52 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:12:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9634T601&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229634T601%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001494669%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/acetylene-torches already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:00 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~210-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clamps/clamp-type~handscrew/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=50415K53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2250415K53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23520 pages (at 46 pages/min), scraped 702 items (at 2 items/min) 2025-11-02 06:13:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~3l160/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clevis-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9634T601&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229634T601%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/rod-ends-and-linkages/super-swivel-ball-joint-rod-ends-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clevis-pins-for-electric-actuators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/acetylene-torches already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~210-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~3-95-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamps/clamp-type~handscrew&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3l160-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000096222%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001549595%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9791062983264&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9791062983264%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001930558%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-tubs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~3l200/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cords/voltage~7-5v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9634T601&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229634T601%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:40 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:13:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A277&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A277%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5008A51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225008A51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~4-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~1-652/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-tubs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3l200-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cords/voltage~7-5v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/amp-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:13:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A277&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A277%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~46-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9957431150989&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9957431150989%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553655%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clamps/material~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~4-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23565 pages (at 45 pages/min), scraped 704 items (at 2 items/min) 2025-11-02 06:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~25-042-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~5-2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~1-652&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70235K245&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270235K245%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=amp-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000135348%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A277&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A277%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001499863%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~46mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamps/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~25-042mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~5-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8989K368&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228989K368%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/for-housing-id~46mm/ 2025-11-02 06:14:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/clamps/material~iron/ 2025-11-02 06:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/casters/environment~washdown/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/din-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:14:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~25-042mm/ 2025-11-02 06:14:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~5-2-mm/ 2025-11-02 06:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/for-use-with~ammonium-hydroxide/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rolling-stools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/acme-threaded-rod-lubricants/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/box-tracks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/for-use-with~bases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~4-7-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=casters/environment-1~washdown&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=din-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1322029403092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1322029403092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001533250%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/casters/environment-1~washdown/ 2025-11-02 06:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~ammonium-hydroxide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acme-threaded-rod-lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=box-tracks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23608 pages (at 43 pages/min), scraped 706 items (at 2 items/min) 2025-11-02 06:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-cleaner-hoods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~bases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968052%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~52-ga-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A294&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A294%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~73-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ball-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179297%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3441N31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223441N31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~75-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/for-use-with~compressed-gas/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~52-ga&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-cleaner-hoods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5542K51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225542K51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~73mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/din-rail-stops/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/roof-drains/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~52-ga/ 2025-11-02 06:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~73mm/ 2025-11-02 06:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~75mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~br1216/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~compressed-gas&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/voltage~18000v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-conditioning-taps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/class-5-steel-square-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=din-rail-stops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roof-drains&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~75mm/ 2025-11-02 06:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969110%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23648 pages (at 40 pages/min), scraped 709 items (at 3 items/min) 2025-11-02 06:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=br1216-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/voltage~18000v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-conditioning-taps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=class-5-steel-square-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7918A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227918A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~3l640/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-caps/specifications-met~asme-b31-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003552960%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6974173590687&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20caps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6974173590687%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002095111%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/material~cobalt-steel/shank-type~morse-taper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bit-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brackets/bracket-type~45-outside-angle/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9634T401&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229634T401%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cutters/for-use-on~inconel-nickel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3l640-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000065001%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/for-use-with~grain-alcohol/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-sets-11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/bracket-type~45-outside-angle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buckets/capacity~110-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutters/for-use-on~inconel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/roof-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/snap-ring-pliers/tip-shape~flat/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/drill-bit-sets-11/ 2025-11-02 06:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561627%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cutters/for-use-on~inconel/ 2025-11-02 06:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/casters/caster-swivel-construction~kingpinless/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-pressure-check-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/corrosion-preventers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=33125T45&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2233125T45%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/direct-drive-mixers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity-weight-~110-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roof-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/class-rk5-fuse-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23692 pages (at 44 pages/min), scraped 711 items (at 2 items/min) 2025-11-02 06:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eaton-style-snap-ring-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/buckets/capacity-weight-~110-gal-/ 2025-11-02 06:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=casters/swivel-construction~kingpinless&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearing-balls/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ball-point-markers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corrosion-preventers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/armored-cable-type~mc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/3m-4910-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=direct-drive-mixers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/casters/swivel-construction~kingpinless/ 2025-11-02 06:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/cable-designation~mtw/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=class-rk5-fuse-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rope-cleats/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-balls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-point-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hook-and-loop-cable-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/industry-designation~mc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~3-402/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3m-4910-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/industry-designation~mtw&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/corrugated-sleeving/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire/industry-designation~mc/ 2025-11-02 06:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rope-cleats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:17:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire/industry-designation~mtw/ 2025-11-02 06:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~35-01-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rope-measurers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/shaft-diameter~1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/hook-and-loop-cable-ties-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clean-room-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~3-402&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buna-n-o-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23732 pages (at 40 pages/min), scraped 712 items (at 1 items/min) 2025-11-02 06:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corrugated-sleeving&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/anchors/hook-and-loop-cable-ties-7/ 2025-11-02 06:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493808%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clean-room-curtains/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~35-01mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~5-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-shaft-diameter~1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rope-measurers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clean-room-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89965K424&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289965K424%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buna-n-o-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~35-01mm/ 2025-11-02 06:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/for-use-on~masonry/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/for-shaft-diameter~1-8/ 2025-11-02 06:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clean-room-curtains&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coatings/for-use-on~cpvc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buna-n-rubber-sheet-gaskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/for-use-on~nickel-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/for-use-with~propane/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~5-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buna-n-rubber-sheet-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cotton-fabric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/for-use-on~masonry&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506618%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~cpvc-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929672%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buna-n-rubber-sheet-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/for-use-on~nickel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/for-use-with~propane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9176T23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229176T23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buna-n-rubber-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cotton-fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-pressure-white-silicone-rubber-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/coatings/for-use-on~cpvc-plastic/ 2025-11-02 06:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23779 pages (at 47 pages/min), scraped 713 items (at 1 items/min) 2025-11-02 06:19:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/for-use-on~nickel/ 2025-11-02 06:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/egg-crate-foam/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cv-joints/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-ball-knobs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/banding-sealers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buckets/for-shipping-contents~hazardous-solids/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cleaned-and-bagged-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~cx144/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~9-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=egg-crate-foam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cv-joints&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-ball-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=banding-sealers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/for-shipping-contents~hazardous-solids&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cleaned-and-bagged-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cx144-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~9-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=104738336527&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B104738336527%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002242093%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cleaners/for-use-on~carbide/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:19:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~9-5-mm/ 2025-11-02 06:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rotary-paper-trimmers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=40015T61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2240015T61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/tube-od~1-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cat-tracks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-strength-white-pvdf-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23820 pages (at 41 pages/min), scraped 714 items (at 1 items/min) 2025-11-02 06:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bur-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~9mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cleaners/for-use-on~carbide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-paper-trimmers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/shaft-diameter~20-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/air-hose/id~9mm/ 2025-11-02 06:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-brass-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/for-tube-od~1-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9998225260164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9998225260164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cat-tracks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493348%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bur-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/adapters/for-tube-od~1-1-8/ 2025-11-02 06:20:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-powered-scissors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=55335K67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2255335K67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-shaft-diameter~20mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-brass-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/shaft-diameter~3-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/burlap-packaging/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-temperature-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/for-shaft-diameter~20mm/ 2025-11-02 06:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/number-of-poles~2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-powered-scissors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbow-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cold-temperature-liners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/length~17-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-shaft-diameter~3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=burlap-packaging&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/belt-drives/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~1304/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23864 pages (at 44 pages/min), scraped 716 items (at 2 items/min) 2025-11-02 06:21:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/number-of-poles~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/for-shaft-diameter~3-16/ 2025-11-02 06:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbow-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cleaners/for-use-on~titanium-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cylinder-rod-couplers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-point-type~standard/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/length~2-29-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001526969%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=belt-drives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/burs/shape~flat-top-cone/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~1304&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cleaners/for-use-on~titanium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylinder-rod-couplers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/point-style~standard&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/divider-trays/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~2-29-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/belt-rollers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002105709%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cleaners/for-use-on~titanium/ 2025-11-02 06:21:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=burs/shape~flat-top-cone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/point-style~standard/ 2025-11-02 06:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001592175%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cylinder-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3908531452254&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3908531452254%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313251%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-shield-blowguns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barbed-hose-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=divider-trays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1077T33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221077T33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=belt-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/size~w/ 2025-11-02 06:22:03 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=burs/shape~flat-top-cone&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 06:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cylinders/body-material~aluminum-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23908 pages (at 44 pages/min), scraped 717 items (at 1 items/min) 2025-11-02 06:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylinder-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-glands/thread-size~m32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~5-63/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:22:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-shield-blowguns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/belt-tensioners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/voltage~12v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barbed-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/shaft-diameter~50-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylinders/body-material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-glands/thread-size~m32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~5-63&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cylinders/body-material~aluminum/ 2025-11-02 06:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/caution-stickers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=belt-tensioners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/voltage~12v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/round-electrical-panel-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6782425493977&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6782425493977%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454704%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498183%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-shaft-diameter~50mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-elbows/fitting-connection~compression-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cylinders/mounting-location~surface/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7310K55&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227310K55%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/for-shaft-diameter~50mm/ 2025-11-02 06:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caution-stickers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~1604/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70235K14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270235K14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-electrical-panel-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/length~39/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barbed-tube-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/tube-od~4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8511825902204&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8511825902204%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361986%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cylinders/mounting-style~surface&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23951 pages (at 43 pages/min), scraped 719 items (at 2 items/min) 2025-11-02 06:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cylinders/mounting-style~surface/ 2025-11-02 06:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/shaft-diameter~7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~1604&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~39&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barbed-tube-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/for-tube-od~4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/length~50-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002093113%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/thread-size~3-4-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-elbows/shape~22-1-2-elbow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001622059%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:23:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/adapters/for-tube-od~4/ 2025-11-02 06:23:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/thread-size~4-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-shaft-diameter~7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334890%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8511825902204&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8511825902204%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310100%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3497N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223497N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/external-power-supply-efficiency-standard~doe-level-vi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~50-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000122836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/for-shaft-diameter~7-8/ 2025-11-02 06:23:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/router-tables/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4880K701&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224880K701%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cad-models&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/air-hose/length~50-ft-/ 2025-11-02 06:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~1-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3497N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223497N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/specifications-met~doe-level-vi-efficiency&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~0-0995/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cad-models&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clevis-pins/diameter~1-2-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=router-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/gender~male-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-vacuum-mufflers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire/specifications-met~doe-level-vi-efficiency/ 2025-11-02 06:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 23998 pages (at 47 pages/min), scraped 720 items (at 1 items/min) 2025-11-02 06:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cad-models&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~1-5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3497N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223497N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-0995&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 500 Internal Server Error 2025-11-02 06:24:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:18 [scrapy.spidermiddlewares.httperror] (PID: 148) INFO: Ignoring response <500 https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cad-models&srchtxt=&mixedworld=force&workstationdata=true>: HTTP status code is not handled or not allowed 2025-11-02 06:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clevis-pins/diameter~1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bushings/for-shaft-diameter~1-5mm/ 2025-11-02 06:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~1623-2rs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/gender~male&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000148009%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~3vx1180/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-drive-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/clevis-pins/diameter~1-2/ 2025-11-02 06:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/shape~hex/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/adapters/gender~male/ 2025-11-02 06:24:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:24:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/document-protectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/thread-size~m11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~1623-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cellulose-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alcohol/clean-room-standard~iso-class-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/d-sub-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/shape~wye/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3vx1180-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454214%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002102769%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/shape~hex&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=document-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cellulose-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alcohol/environmental-rating~iso-class-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=d-sub-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000141509%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/pipe-size~1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~14-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24043 pages (at 45 pages/min), scraped 720 items (at 0 items/min) 2025-11-02 06:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~1-11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/shape~wye&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/alcohol/environmental-rating~iso-class-5/ 2025-11-02 06:25:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dog-clips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~3vx355/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/beryllium-copper-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/countersinks/for-use-on~tool-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/countersinks/for-use-on~wood-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310158%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/pipe-size~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~14mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/drive-size~1-11-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dog-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086699%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bushings/for-shaft-diameter~14mm/ 2025-11-02 06:25:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/combination-wrenches/drive-size~1-11-16-1/ 2025-11-02 06:25:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3vx355-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=beryllium-copper-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/for-use-on~tool-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/for-use-on~wood&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barricade-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~d371/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=983458193638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B983458193638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/countersinks/for-use-on~wood/ 2025-11-02 06:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clevis-pins/diameter~6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/all-weather-hydraulic-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/drive-size~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cement/for-use-on~galvanized-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-055/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barricade-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=d371-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24085 pages (at 42 pages/min), scraped 720 items (at 0 items/min) 2025-11-02 06:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3236610467807&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=combination%20wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3236610467807%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489937%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~8-01-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-voltage-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clevis-pins/diameter~6mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=all-weather-hydraulic-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/clevis-pins/diameter~6mm/ 2025-11-02 06:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-055&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~1657-2z/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barriers/material~leather-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~0-23-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cleaners/for-use-on~plastic-1/for-use-on~abs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~8-01mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001495604%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cement/for-use-on~plastic-1/for-use-on~pvdf/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~8-01mm/ 2025-11-02 06:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~1657-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barriers/material~leather&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~0-23mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cleaners/for-use-on~abs-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-alloy-260-shim-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001495604%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002113893%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/barriers/material~leather/ 2025-11-02 06:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~1654-2rs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/shank-size~0-23mm/ 2025-11-02 06:26:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cleaners/for-use-on~abs-plastic/ 2025-11-02 06:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cement/for-use-on~pvdf-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~9-01-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barriers/material~zinc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-alloy-260-shim-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cement/for-use-on~pvdf-plastic/ 2025-11-02 06:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bin-box-storage-cabinets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24129 pages (at 44 pages/min), scraped 720 items (at 0 items/min) 2025-11-02 06:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001495604%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rulers/length~0-7-m/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~1654-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cement-cutting-drill-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinge-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~9-01mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barriers/material~zinc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002080910%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bin-box-storage-cabinets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/material~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~0-7-m&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~9-01mm/ 2025-11-02 06:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9998883551574&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9998883551574%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512333%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002892372%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~36-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coatings/for-use-on~rubber-2/for-use-on~epdm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/contact-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-and-cable-crimper-dies/for-wire-gauge~350-mcm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:46 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~0-7-m&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 06:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/center-drills/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-chuck-key-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/length~1-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1271809236341&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20and%20cable%20crimper%20dies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1271809236341%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002000721%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/elbows/material~iron/ 2025-11-02 06:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-128/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:00 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:28:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~epdm-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-1285/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4972N41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224972N41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24173 pages (at 44 pages/min), scraped 720 items (at 0 items/min) 2025-11-02 06:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rulon-j-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-pipe-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/coatings/for-use-on~epdm-rubber/ 2025-11-02 06:28:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ansi-ladders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-128&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001591770%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-1285&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulon-j-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496507%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/lead-screws-and-screw-jacks/self-reversing-lead-screws-and-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/fitting-type~union-1/for-use-with~steam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497183%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ansi-ladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clips/finish~phosphate-coated/color~black-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001919079%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968293%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3671N121&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223671N121%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/days-without-accident-signs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-1564/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~0-1564/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-extrusion-casters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:52 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~4l160/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:28:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bushings/for-shaft-diameter~5-16/ 2025-11-02 06:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clips/finish~black-phosphate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rv-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/entry-door-hinges/pin-type~lift-off/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=days-without-accident-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-1564&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/shaft-diameter~13-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24215 pages (at 42 pages/min), scraped 722 items (at 2 items/min) 2025-11-02 06:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-1564&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-extrusion-casters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/clips/finish~black-phosphate/ 2025-11-02 06:29:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=4l160-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497058%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000133010%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/length~12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rv-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/lift-off-entry-door-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/door-louvers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-shaft-diameter~13mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~19-mm-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~1-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%221510659569138%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-sleeves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/couplings/for-shaft-diameter~13mm/ 2025-11-02 06:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~80-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~1-1-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1849A62&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221849A62%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-louvers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-1887/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~19-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001622323%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~80mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:29:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/combination-wrenches/size~19-mm/ 2025-11-02 06:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~1-1mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000018197%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~3003-2z/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-1887&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bushings/for-shaft-diameter~80mm/ 2025-11-02 06:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24259 pages (at 44 pages/min), scraped 723 items (at 1 items/min) 2025-11-02 06:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-socket-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/antifriction-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/shank-size~1-1mm/ 2025-11-02 06:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-194/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506167%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-196/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/antiseize-lubricating-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bit-extenders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~3003-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~9mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=antifriction-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002142227%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bushings/for-shaft-diameter~9mm/ 2025-11-02 06:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/door-slides/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/material~copper-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-194&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-196&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=antiseize-lubricating-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bit-extenders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/length~2-3-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001501931%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497385%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:30:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/deadman-ball-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-rope-sleeves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-slides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/material~copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9473N249&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229473N249%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/deburring-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=96511A659&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2296511A659%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:30:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~2-3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-discs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bushings/material~copper/ 2025-11-02 06:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=deadman-ball-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24304 pages (at 45 pages/min), scraped 725 items (at 2 items/min) 2025-11-02 06:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-rope-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cable-ties/length~2-3-16/ 2025-11-02 06:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-door-plates/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/thread-type~metric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=deburring-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-wall-mount-ladders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/thread-size~pg-11/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-discs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-and-cable-strippers/cable-designation~single-mode-os2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/deburring-wheel-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-weld-pipe-elbows/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-door-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/thread-type~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=506136630644&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20and%20cable%20strippers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B506136630644%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003015757%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/thread-size~1-1-2-6-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-wall-mount-ladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~pg-11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ceramic-standoffs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070699%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4827N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224827N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=deburring-wheel-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454405%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-weld-pipe-elbows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~1-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~1-1-2-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/doors/material~wood-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7805K71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227805K71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-251/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ceramic-standoffs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bushings/thread-size~1-1-2-6/ 2025-11-02 06:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/thread-size~1-1-8-7-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/safety-lanyards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24346 pages (at 42 pages/min), scraped 727 items (at 2 items/min) 2025-11-02 06:32:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/size~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=doors/material~wood&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/closed-end-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/crowfeet/size~1-1-2/ 2025-11-02 06:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-251&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~1-7-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/size~1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/doors/material~wood/ 2025-11-02 06:32:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~1-1-8-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/thread-type~nh-nst/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-lanyards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496084%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/crowfeet/size~1-2/ 2025-11-02 06:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/size~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bushings/thread-size~1-1-8-7/ 2025-11-02 06:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=closed-end-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/size~1-7-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/crowfeet/size~1-4/ 2025-11-02 06:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~0-6262/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~0-6255/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dot-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-type~nh-nst&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/crowfeet/size~1-7-16/ 2025-11-02 06:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metric-steel-precision-acme-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drilled-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bladders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/overall-length~30-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038314%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/delay-timers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-6262&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dot-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-6255&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6523T51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226523T51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cga-500-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-steel-precision-acme-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24391 pages (at 45 pages/min), scraped 727 items (at 0 items/min) 2025-11-02 06:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~30-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070600%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000107009%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=delay-timers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001604457%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/for-use-with~ethylene-glycol-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:33:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:33:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~0-687/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/overall-length~36-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/anchor-rope/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cga-500-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3097T85&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223097T85%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-319/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~0-7535/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/roller-chain-and-links/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~ethylene-glycol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-687&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~36-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~4l970/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchor-rope&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-319&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pipe-fittings/for-use-with~ethylene-glycol/ 2025-11-02 06:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~0-7535&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/extended-life-lightweight-ansi-roller-chain-and-links&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000043291%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8347067787852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8347067787852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001630102%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:33:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~3-8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496691%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000012110%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain/extended-life-lightweight-ansi-roller-chain-and-links/ 2025-11-02 06:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3097T44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223097T44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-component-knobs/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=4l970-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24437 pages (at 46 pages/min), scraped 729 items (at 2 items/min) 2025-11-02 06:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038489%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9473N329&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229473N329%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-brush-handle-and-extension-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-ties/overall-length~5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~2-55-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=13615A56&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2213615A56%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/detail-cleaning-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/salt-spreaders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-machine-component-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/material~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/thread-size~3-8-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-ties/length~5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~2-55mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/double-seal-o-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~4-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/cable-ties/length~5-8/ 2025-11-02 06:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=detail-cleaning-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=salt-spreaders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/shank-size~2-55mm/ 2025-11-02 06:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/salt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~3-8-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~6000-2z/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000158576%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-brushes/handle-material~wood-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain/material~iron/ 2025-11-02 06:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454138%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=double-seal-o-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1528A21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221528A21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesive-back-strips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=salt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24479 pages (at 42 pages/min), scraped 732 items (at 3 items/min) 2025-11-02 06:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95365A111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295365A111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6000-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-brushes/handle-material~wood&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=computer-cable-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alloy-steel-hollow-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1528A21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221528A21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-back-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire-brushes/handle-material~wood/ 2025-11-02 06:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~2mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/for-use-with~helium/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesive-dispensing-tips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/shank-size~2mm/ 2025-11-02 06:35:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~6005-2rs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alloy-steel-hollow-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1528A21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221528A21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/for-use-with~hydrogen-iodide/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~2014/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-cable-trays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~helium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-dispensing-tips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~pharmaceuticals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964764%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6005-2rs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cm-hoists/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:35:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:36:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968243%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~2014&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002086088%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24526 pages (at 47 pages/min), scraped 733 items (at 1 items/min) 2025-11-02 06:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/thread-size~7-8-9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hammers/manufacturer~dewalt-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alumina-ceramic-powders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002097600%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4759K211&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224759K211%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cm-hoists&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7697K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227697K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~2459154-0007/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~7-8-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/overall-width~7-9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dewalt-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alumina-ceramic-powders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~0-75-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333387%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000094969%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mica-rod-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~2459154-0007&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:48 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/push-to-connect-tube-fittings/material~brass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1264A13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221264A13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sanding-sticks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~1-15-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~0-75-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116568%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hinges/overall-width~9-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mica-rod-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:36:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~0-75-mm/ 2025-11-02 06:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2286N29&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222286N29%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/for-use-on~galvanized-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-push-to-connect-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanding-sticks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24566 pages (at 40 pages/min), scraped 736 items (at 3 items/min) 2025-11-02 06:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-15-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~1-25-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000016687%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/10-chain/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hinges/overall-width~9-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~45-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum/material~aluminum-2/material~6020-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~galvanized-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~6-7-fl-oz/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~1-25-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930477%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-shafts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/trade-size~10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/calipers/manufacturer-model-number~02azd730g-c/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~45-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~1-25-mm/ 2025-11-02 06:37:40 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum/material~aluminum/material~6020-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~6-7-oz-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain/trade-size~10/ 2025-11-02 06:37:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000098941%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/for-use-on~magnets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510688%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/containers/capacity-weight-~6-7-oz-/ 2025-11-02 06:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/for-use-with~pharmaceuticals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-isolators/environment~sanitary/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/manufacturer-model-number~02azd730g-c&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~5-3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/sash-chain-trade-size~25/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8826536049195&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8826536049195%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002098029%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~5-75-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/asme-hex-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~magnets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24606 pages (at 40 pages/min), scraped 737 items (at 1 items/min) 2025-11-02 06:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9038K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229038K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-gauge-isolators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~5-3mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/trade-size~25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~16-0-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/shank-size~5-3mm/ 2025-11-02 06:38:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=asme-hex-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~6200/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain/trade-size~25/ 2025-11-02 06:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/sash-chain-trade-size~45/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~1-378/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~6202/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~16-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000168713%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit/conduit-type~emt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6200&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/trade-size~45&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-378&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~16-mm/ 2025-11-02 06:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6202&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/material~plastic-1/material~cpvc/pipe-schedule~80&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5860541886895&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conduit&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5860541886895%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001566963%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092551%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:38:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain/trade-size~45/ 2025-11-02 06:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000127939%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002125839%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain/straight-link-chain-trade-size~7-0/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-square-head-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001545827%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~2/ 2025-11-02 06:39:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24653 pages (at 47 pages/min), scraped 738 items (at 1 items/min) 2025-11-02 06:39:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/boat-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/micrometer-cases/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~5006/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~3-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/material~plastic-1/pressure-class~150&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain/trade-size~7-0&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-square-head-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/formulation~aliphatic-resin/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-clips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000127660%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain/trade-size~7-0/ 2025-11-02 06:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=boat-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=micrometer-cases&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~5006&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/formulation~aliphatic-resin&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/assembly-grease/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126442%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-grade-plastics/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit-box-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002087492%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-connectors/number-of-poles~15/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6762K46&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226762K46%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~1-551/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hitch-inserts-for-forklift-attachments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7476A16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227476A16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=assembly-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24697 pages (at 44 pages/min), scraped 739 items (at 1 items/min) 2025-11-02 06:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-grade-plastics&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/formulation~calcium-carbonate/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-connectors/number-of-poles~15-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~6209/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-551&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/pipe-size~1-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7476A16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227476A16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire-connectors/number-of-poles~15-1/ 2025-11-02 06:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000149124%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dial-indicators/manufacturer-model-number~513-403-10e/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/formulation~calcium-carbonate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/shank-size~8-4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6209&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/pipe-size~1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7197N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227197N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108892%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7476A16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227476A16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002127518%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-indicators/manufacturer-model-number~513-403-10e&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000156884%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4846N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224846N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/shank-size~8-4mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133854%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~4-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolt-clamp-hose-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:40:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/shank-size~8-4mm/ 2025-11-02 06:40:59 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~2-15-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/formulation~calcium-sulfate-gypsum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coatings/for-use-on~rubber-2/for-use-on~viton-fluoroelastomer/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-thumb-screw-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24742 pages (at 45 pages/min), scraped 741 items (at 2 items/min) 2025-11-02 06:41:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/5v800-v-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~6am6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133854%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~4-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolt-clamp-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/drive-size~2-15-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~4-5-8/ 2025-11-02 06:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/formulation~gypsum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~viton-fluoroelastomer-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-thumb-screw-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5v800-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/drive-size~2-15-16-1/ 2025-11-02 06:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~6am6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-6885/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260072862737&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260072862737%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133854%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/adhesives/formulation~gypsum/ 2025-11-02 06:41:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:41:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/coatings/for-use-on~viton-fluoroelastomer-rubber/ 2025-11-02 06:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~2-5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/thread-size~pg-48/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002141377%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/5v950-v-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~6fc52470aa180aa0/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~1-631/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/calipers/manufacturer-model-number~500-753-20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71455K56&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271455K56%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-6885&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:52 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/drive-size~2-5-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~pg-48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:41:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498629%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5v950-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolt-handles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~6fc52470aa180aa0&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/drive-size~2-5-16-1/ 2025-11-02 06:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~1-631&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/manufacturer-model-number~500-753-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24788 pages (at 46 pages/min), scraped 742 items (at 1 items/min) 2025-11-02 06:42:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454841%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~5vx1150/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/job-saws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002090042%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000076474%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8826536049195&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8826536049195%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121249%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=69915K132&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2269915K132%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brass-u-channels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolt-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2286N38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222286N38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5vx1150-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit-knockouts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saws/folding-pocket-handsaws-for-wood&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-057/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/astm-b355-copper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-u-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-links/roller-chain-trade-number~10b/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=661583884380&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B661583884380%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%225237127425656%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/apple-iphone-compatible-accessories/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-bars/material~aluminum-2/material~6013-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/butt-weld-pipe-tees/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8635A12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228635A12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-057&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=astm-b355-copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~1-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-links/roller-chain-trade-size~10b-ss&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001540169%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=apple-iphone-compatible-accessories&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-bars/material~aluminum/material~6013-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hoes/blade-material~aluminum-bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24833 pages (at 45 pages/min), scraped 745 items (at 3 items/min) 2025-11-02 06:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=butt-weld-pipe-tees&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain-links/roller-chain-trade-size~10b-ss/ 2025-11-02 06:43:11 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-057&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 06:43:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aqueous-cleaning-solutions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~1-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9990216947512&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9990216947512%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511688%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6766560220190&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hoes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6766560220190%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539862%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/butt-weld-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-073/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~1-3-4/ 2025-11-02 06:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mig-welding-gun-tips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9050K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229050K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sb-an-batteries/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6537A14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226537A14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aqueous-cleaning-solutions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/breadboards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=butt-weld-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-073&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mig-welding-gun-tips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/bearing-trade-number~6304-2z/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sb-an-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6537A14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226537A14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conveyor-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001526682%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=breadboards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-links/roller-chain-trade-number~40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:43:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/aluminum-bars/material~aluminum/tolerance-rating~tight already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:00 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-073&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 06:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6304-2z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/arbor-press-die-shoes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-084/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24875 pages (at 42 pages/min), scraped 746 items (at 1 items/min) 2025-11-02 06:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002120029%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collar-spacers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-links/roller-chain-trade-size~40-ss&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6537A14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226537A14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=confined-space-ventilators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-panel-locks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=arbor-press-die-shoes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-084&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain-links/roller-chain-trade-size~40-ss/ 2025-11-02 06:44:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/aluminum-bars/material~aluminum/tolerance-rating~tight already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/thread-size~5-16-18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collar-spacers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/manufacturer~loctite/manufacturer-model-number~pc-7357/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001551017%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~2-15-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-panel-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/button-head-tapping-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:42 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-084&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 06:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/scalpel-knives/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/thread-size~5-16-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:45 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:44:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/manufacturer~loctite/manufacturer-model-number~pc-9599/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~22-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/manufacturer~loctite-/manufacturer-model-number~pc-7357&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~2-15-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/finish~passivated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/breakers/current~35-amp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333458%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=button-head-tapping-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002149716%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scalpel-knives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~2-15-16/ 2025-11-02 06:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/manufacturer~loctite-/manufacturer-model-number~pc-9599&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24918 pages (at 43 pages/min), scraped 746 items (at 0 items/min) 2025-11-02 06:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~22-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1830A12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221830A12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/finish~passivated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/scheduling-boards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=breakers/current~35a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-links/leaf-chain-trade-number~bl634/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002124587%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~22-mm/ 2025-11-02 06:45:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/astm-lag-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:45:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001530500%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/breakers/current~35a/ 2025-11-02 06:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-cam-and-groove-hose-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~0-935/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1830A9&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221830A9%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/diamond-coated-router-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/thread-size~10-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scheduling-boards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-links/roller-chain-trade-size~bl634&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=astm-lag-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-strain-relief/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chain-links/roller-chain-trade-size~bl634/ 2025-11-02 06:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-cam-and-groove-hose-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-channels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diamond-coated-router-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~10-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~125-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/fitting-connection~quick-disconnect/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~12-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-strain-relief&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~0-935&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 24958 pages (at 40 pages/min), scraped 748 items (at 2 items/min) 2025-11-02 06:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/military-specification~ms51575-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/audio-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493559%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~125mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~12mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/fitting-connection~quick-disconnect-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chain-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/holders/finish~carbide-tipped/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89955K328&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289955K328%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~125mm/ 2025-11-02 06:46:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/od~12mm/ 2025-11-02 06:46:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/connectors/fitting-connection~quick-disconnect-1/ 2025-11-02 06:46:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/specifications-met~ms51575-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=audio-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/australia-flags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/diamond-locating-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mil-spec-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bolts/specifications-met~ms51575-16/ 2025-11-02 06:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chain-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mil-spec-electrical-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=holders/material~carbide-tipped-hardened-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct/component~airflow-damper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-drilling-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/holders/material~carbide-tipped-hardened-steel/ 2025-11-02 06:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~ep12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=australia-flags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diamond-locating-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mil-spec-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/8-pin-relays/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mil-spec-electrical-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chair-rails/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:46:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/component~airflow-damper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-drilling-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25001 pages (at 43 pages/min), scraped 749 items (at 1 items/min) 2025-11-02 06:47:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-ductwork/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/drive-size~ep12-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002262697%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cam-and-groove-hose-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=8-pin-relays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chair-rails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/drive-size~ep12-1/ 2025-11-02 06:47:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/800-8m-timing-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/astm-a108-threaded-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-ductwork&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duo-cone-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pins/diameter~0-143/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mil-spec-lock-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~1-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cam-and-groove-hose-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-diggers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=800-8m-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~ip20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~d186a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=astm-a108-threaded-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/diameter~0-143&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mil-spec-lock-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~1-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stop-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:47:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~bx50/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:48:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548438%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolts/thread-size~00-96/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/drive-size~ip20-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25042 pages (at 41 pages/min), scraped 749 items (at 0 items/min) 2025-11-02 06:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/manufacturer-equivalent-number~d186a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screw-conveyors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/thread-size~9-16-18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89955K659&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289955K659%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drive-sockets/drive-size~ip20-1/ 2025-11-02 06:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-switch-stops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108556%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bx50-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548438%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~00-96&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-extrusion-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7604K35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227604K35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screw-conveyors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~9-16-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451073%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drilling-hammers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000071717%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8905842209266&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8905842209266%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548438%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-extrusion-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=british-standard-whitworths&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cordless-telephone-batteries/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/shape~180-bend&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mil-spec-stainless-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drilling-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fasteners/fastener-head-type~through-hole/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/shape~straight&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310952%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:49:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdriver-bit-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdriver-power-bit-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:49:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25089 pages (at 47 pages/min), scraped 751 items (at 2 items/min) 2025-11-02 06:49:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000124163%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mil-spec-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~0-033/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-punches/hole-diameter~21-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/head-type~through-hole&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdriver-bit-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/thread-type~npt/pipe-size~1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdriver-power-bit-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=container-dollies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/fasteners/head-type~through-hole/ 2025-11-02 06:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~0-069/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000033831%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~0-0330&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-punches/hole-diameter~27-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/for-hole-diameter~21mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/containers/capacity~3-gal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-mesh/material~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flat-head-screw-assortments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/screwdrivers/size~0-0330/ 2025-11-02 06:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~0-168/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hole-punches/for-hole-diameter~21mm/ 2025-11-02 06:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~0-0690&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/milling-setup-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000033831%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/for-hole-diameter~27mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/backpack-misters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=containers/capacity-weight-~3-gal-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-mesh/material~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:49:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/screwdrivers/size~0-0690/ 2025-11-02 06:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flat-head-screw-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~0-1680&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hole-punches/for-hole-diameter~27mm/ 2025-11-02 06:50:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/containers/capacity-weight-~3-gal-/ 2025-11-02 06:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25133 pages (at 44 pages/min), scraped 751 items (at 0 items/min) 2025-11-02 06:50:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wire-mesh/material~stainless-steel/ 2025-11-02 06:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/food-grade-buckets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=milling-setup-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/screwdrivers/size~0-1680/ 2025-11-02 06:50:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000033831%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/backup-power-supplies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=backpack-misters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/food-industry-thermometers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-punches/hole-diameter~8-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-belt-lacers/lacing-number~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-grade-buckets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:34 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/framing/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=backup-power-supplies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5722855303070&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20belt%20lacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5722855303070%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000069353%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-industry-thermometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/for-hole-diameter~8-5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-belt-lacers/lacing-number~2sp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6139K4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226139K4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/garden-hose-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hole-punches/for-hole-diameter~8-5mm/ 2025-11-02 06:50:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5722855303070&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20belt%20lacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5722855303070%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000023268%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-belt-lacers/lacing-number~36ll&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing/material~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/hole-diameter~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~ip8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6139K2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226139K2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5722855303070&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20belt%20lacers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5722855303070%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000019660%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/framing/material~brass/ 2025-11-02 06:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/hole-diameter~1-9-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3430402569734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3430402569734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001670259%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25181 pages (at 48 pages/min), scraped 751 items (at 0 items/min) 2025-11-02 06:51:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6139K4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226139K4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=garden-hose-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-rope-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6139K5&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226139K5%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glass-filled-nylon-socket-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3430402569734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3430402569734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002234768%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~ip8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6139K2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226139K2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6139K4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226139K4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ball-transfer-conveyors-and-stands/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/miniature-idler-rollers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3430402569734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3430402569734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001670259%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/screwdrivers/size~ip8/ 2025-11-02 06:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/goo-gone/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass-filled-nylon-socket-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6139K2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226139K2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:51:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:51:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-feet&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-transfer-conveyors-and-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miniature-idler-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3430402569734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3430402569734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001670259%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/hole-diameter~34-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/scribers/power-source~electric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=goo-gone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:51:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3430402569734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3430402569734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243577%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:00 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:52:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/band-saw-stands/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/band-saw-blade-guards already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25229 pages (at 48 pages/min), scraped 752 items (at 1 items/min) 2025-11-02 06:52:08 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bar-stock-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scribers/power-source~electric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants-with-material-certification/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/hole-diameter~47-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-shelving-card-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/minimal-spill-hose-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8496026489774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8496026489774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076416%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=band-saw-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3430402569734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3430402569734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002250325%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bar-stock-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-splices/splice-type~butt-splice&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants-with-material-certification&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/band-saw-blade-guards already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barrel-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=minimal-spill-hose-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/barrel-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9877454814573&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20splices&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9877454814573%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000008402094%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/security-door-viewers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-stripper-crimpers/for-wire-gauge~16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/misters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barrel-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:56 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=barrel-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:52:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/self-centering-drill-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-gaskets/material~304-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=security-door-viewers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9324322780279&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wire%20stripper%20crimpers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9324322780279%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002003446%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drop-legs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25273 pages (at 44 pages/min), scraped 752 items (at 0 items/min) 2025-11-02 06:53:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9811964018420&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9811964018420%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510515%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=misters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=self-centering-drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-gaskets/pipe-size~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=44955K151&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2244955K151%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drop-warp-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-gaskets/pipe-size~2-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9811964018420&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9811964018420%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086699%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cord-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-tie-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mitutoyo-compatible-indicator-backs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drum-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9811964018420&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9811964018420%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000104826%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mitutoyo-digital-micrometers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-tips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mitutoyo-compatible-indicator-backs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hook-and-loop-straps/material~polyester-fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/semiconductor-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~4-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6020751333810&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hook%20and%20loop%20straps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6020751333810%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512635%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~4-mm/ 2025-11-02 06:54:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-tube-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/battery-connector-lockouts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25320 pages (at 47 pages/min), scraped 753 items (at 1 items/min) 2025-11-02 06:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/battery-crimping-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mitutoyo-digital-micrometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=semiconductor-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~5-8/ 2025-11-02 06:54:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-hangers/finish~ppa-coated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-walls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mitutoyo-spc-cable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/serialized-zip-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-connector-lockouts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-hangers/material~foam-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-crimping-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5866438711872&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20hangers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5866438711872%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542628%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hook-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mitutoyo-spc-cable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=serialized-zip-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hooks/application~for-lifting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-hangers/material~foam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hooks/attachment-type~plain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/set-screw-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6392083707491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hooks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6392083707491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002012536%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pipe-hangers/material~foam/ 2025-11-02 06:54:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hooks/attachment-type~sew-on&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:54:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hooks/finish~brass-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-saws/diameter~1-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6392083707491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hooks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6392083707491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003934188%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wiring-trough-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mixers/for-use-with~soluble-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/setup-clamp-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6392083707491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hooks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6392083707491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010731%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6392083707491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hooks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6392083707491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001544756%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25368 pages (at 48 pages/min), scraped 753 items (at 0 items/min) 2025-11-02 06:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=set-screw-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-saws/diameter~1-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3648T45&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223648T45%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sewer-filters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-saws/diameter~11-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mixers/for-use-with~soluble-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=setup-clamp-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:55:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-laterals/material~polyethylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000073039%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8018988495419&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8018988495419%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179259%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~1-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/beaded-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fencing/material~wood-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sewer-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~11mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4066A27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224066A27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9977K6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229977K6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000010007%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shaft-mesh-protectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hole-saws/diameter~11mm/ 2025-11-02 06:55:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4066A32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224066A32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-fencing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=beaded-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-laterals/material~polyethylene-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearing-gauges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=core-pins/fastener-head-type~flush-mount&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4066A27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224066A27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearing-grease-packers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pipe-laterals/material~polyethylene-plastic/ 2025-11-02 06:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-mesh-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25414 pages (at 46 pages/min), scraped 756 items (at 3 items/min) 2025-11-02 06:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4352758098895&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=core%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4352758098895%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001591322%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:18 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=core-pins/fastener-head-type~flush-mount&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 06:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4066A27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224066A27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-grease-packers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=horsehair-wheel-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/color~light-gray&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corner-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corner-strut-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~150-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002200122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:38 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-screws/drive-size~5-32-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shell-end-mill-arbors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mobil-velocite-no-3-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6803K71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226803K71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~150mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-saws/for-use-on~titanium-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/bearings/id~150mm/ 2025-11-02 06:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-screws/drive-style~external-hex/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/for-use-with~paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-saws/for-use-on~ceramic-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shell-end-mill-arbors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965550%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-screws/drive-size~5-32-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/for-use-on~titanium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shims/thickness~1-750-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25456 pages (at 42 pages/min), scraped 757 items (at 1 items/min) 2025-11-02 06:57:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-screws/length~140-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7859T71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227859T71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-nipples-and-pipe/length~1-1-2-ft/length~18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/for-use-on~ceramic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hole-saws/for-use-on~titanium/ 2025-11-02 06:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-screws/drive-style~external-hex/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2724637663046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wood%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2724637663046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001594091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=525781165543&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20nipples%20and%20pipe&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B525781165543%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001624697%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001629911%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hole-saws/for-use-on~ceramic/ 2025-11-02 06:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:24 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:57:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-screws/length~60-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:57:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/thickness~1-75mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97498A151&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297498A151%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shock-absorbing-decelerators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=counters/mounting-location~panel-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jackets/material~leather-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2724637663046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wood%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2724637663046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001616508%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shims/thickness~1-75mm/ 2025-11-02 06:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=525781165543&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20nipples%20and%20pipe&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B525781165543%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001624697%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001629911%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-049&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:57:37 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:57:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5519429302621&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=counters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5519429302621%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002253612%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shock-absorbing-decelerators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497443%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/countersinks/for-use-on~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jackets/material~leather&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-nipples-and-pipe/length~5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mobilux-ep-0-grease/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-screws/screw-size~m12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=525781165543&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20nipples%20and%20pipe&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B525781165543%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001624697%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001629911%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8988K27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228988K27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5519429302621&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=counters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5519429302621%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002253612%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:57:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/jackets/material~leather/ 2025-11-02 06:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=525781165543&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20nipples%20and%20pipe&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B525781165543%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001607234%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2724637663046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wood%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2724637663046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334339%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersinks/for-use-on~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25505 pages (at 49 pages/min), scraped 760 items (at 3 items/min) 2025-11-02 06:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobilux-ep-0-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97498A171&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297498A171%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:12 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-screws/screw-size~no-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/countersinks/for-use-on~aluminum/ 2025-11-02 06:58:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bend-close-d-ring-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5519429302621&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=counters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5519429302621%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002253612%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2724637663046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wood%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2724637663046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334678%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-082&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/bend-and-stay-air-hose already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wood-screws/drive-size~t30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-screws/screw-size~no-9-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001492765%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dry-core-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bend-close-d-ring-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:38 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bicycle-flags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-nipples-and-pipe/pipe-size~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2724637663046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wood%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2724637663046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334324%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8988K23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228988K23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wood-screws/size~t30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/bend-and-stay-air-hose already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=525781165543&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20nipples%20and%20pipe&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B525781165543%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000050647%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/additive~nickel-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wood-screws/size~t30/ 2025-11-02 06:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shoulder-straps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bicycle-flags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:58:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2724637663046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wood%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2724637663046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334324%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-158&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/additive~nickel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25551 pages (at 46 pages/min), scraped 762 items (at 2 items/min) 2025-11-02 06:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505450%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:11 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-166&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:11 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 06:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shoulder-straps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lubricants/additive~nickel/ 2025-11-02 06:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~13-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=work-holding-lift-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2724637663046&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wood%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2724637663046%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334324%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6100K311&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226100K311%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001499014%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/for-use-on~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-16inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drywall-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-203125inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-16inches/ 2025-11-02 06:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5560K9&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225560K9%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 06:59:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/for-use-on~roller-chain/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-203125inches/ 2025-11-02 06:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/shaft-diameter~24-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-use-on~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dual-action-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-241/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/momentary-toggle-pilot-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-plugs/for-use-with~sulfuric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002280368%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/lubricants/for-use-on~plastic/ 2025-11-02 06:59:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 06:59:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/silicone-food-grade-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 06:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-use-on~roller-chain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8841143350592&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8841143350592%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/blade-guards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-plugs/pressure-class~150&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-241inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=momentary-toggle-pilot-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25600 pages (at 49 pages/min), scraped 764 items (at 2 items/min) 2025-11-02 07:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930150%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=52195K278&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2252195K278%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8841143350592&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8841143350592%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000127660%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silicone-food-grade-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-241inches/ 2025-11-02 07:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-308&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blade-guards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/component~gfci-receptacle-outlet-box-cover&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/blank-cardstock-colored-tags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493508%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/blank-laminated-cardstock-tags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002265606%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=covid-shields&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K405&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K405%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=workbenches/mobility~mobile/top-material~stainless-steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71865K64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271865K64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/component~outer-corner-90-elbow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blank-cardstock-colored-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cpi-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8316204383934&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=workbenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8316204383934%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000109595%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002209688%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002277192%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-cabinet-workbenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blank-laminated-cardstock-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=workbenches/mounting-location~wall&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K405&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K405%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71865K351&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271865K351%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/single-layer-graphene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8316204383934&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=workbenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8316204383934%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002266240%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25652 pages (at 52 pages/min), scraped 766 items (at 2 items/min) 2025-11-02 07:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/blasting-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/component~security-strap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=workbenches/stainless-steel-cabinet-workbenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/component~stacking-clamp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K405&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K405%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~lb-8065-c5-a-copper-based-anti-seize/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002271430%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=single-layer-graphene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/blind-flange-locks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8316204383934&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=workbenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8316204383934%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%221759242434563%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002268527%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blasting-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=71865K461&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2271865K461%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4139T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224139T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2236T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222236T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:35 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-472&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~lb-8065-c5-a-copper-based-anti-seize&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sintered-breather-vents/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=blind-flange-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496055%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/morse-taper-sleeves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143896%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~lubriplate-syn-lube-68/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/twine/material~sisal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-reducers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=51155K68&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2251155K68%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sintered-breather-vents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~mobil-dte-26-ultra/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-routers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25698 pages (at 46 pages/min), scraped 771 items (at 5 items/min) 2025-11-02 07:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=morse-taper-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143896%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:11 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sisal-twine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~lubriplate-syn-lube-68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrap-insulation&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slant-top-enclosures/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slatwall-hooks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~mobil-dte-26-ultra&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002151025%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-shrouds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/trade-name~mobilith-shc-220/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143896%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002100376%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1016K91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221016K91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/id~0-785/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slant-top-enclosures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-742&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slatwall-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/boiler-gate-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001507175%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~mobilith-shc-220&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002100376%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~1-5-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-tees/material~plastic-1/material~pfa-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-785inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002084043%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=48855K22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2248855K22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=boiler-gate-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25745 pages (at 47 pages/min), scraped 772 items (at 1 items/min) 2025-11-02 07:03:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/id~0-785inches/ 2025-11-02 07:03:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1394K4&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221394K4%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bolt-together-structural-framing-angles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutoff-lathe-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002100376%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/drive-size~1-5-16-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-tees/material~pfa-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-blowers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/drive-size~1-5-16-2/ 2025-11-02 07:03:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pipe-tees/material~pfa-plastic/ 2025-11-02 07:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/motors/voltage~110v-ac-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolt-together-structural-framing-angles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:33 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-threader-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-collars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-downdraft-diverters/duct-size~5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-trucks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/drive-size~46-mm-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=motors/voltage~110v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bottle-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-end-caps/duct-size~18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5471549406771&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct%20downdraft%20diverters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5471549406771%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001992482%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-warmers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=513557860808&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B513557860808%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489952%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/motors/voltage~110v-ac/ 2025-11-02 07:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metal-belleville-spring-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4860285128955&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct%20end%20caps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4860285128955%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001992644%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mounting-flanges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25794 pages (at 49 pages/min), scraped 774 items (at 2 items/min) 2025-11-02 07:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-wrap/flammability-rating~astm-e84-0-0-for-flame-and-smoke&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bottle-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/movie-screens/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/drive-size~e4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4033245586179&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pipe%20wrap&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4033245586179%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092916%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-belleville-spring-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4860285128955&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct%20end%20caps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4860285128955%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001992644%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mounting-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~160-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=513557860808&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B513557860808%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003488433%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-hose/color~white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=45985K31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2245985K31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brake-bleeders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=movie-screens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001506546%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8707661853867&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8707661853867%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4860285128955&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=duct%20end%20caps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4860285128955%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001992644%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-122160-helical-inserts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/military-specification~ms122132/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=53335K71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2253335K71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=513557860808&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B513557860808%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003488433%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brake-bleeders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dc-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/solder-joint-asme-code-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:05:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-122160-helical-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-122132-helical-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/drive-size~t100&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25842 pages (at 48 pages/min), scraped 776 items (at 2 items/min) 2025-11-02 07:05:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=513557860808&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B513557860808%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003488433%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct-reducers/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=513557860808&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=wrenches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B513557860808%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003488394%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=neoprene%252fepdm%252fsbr-rubber-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solder-joint-asme-code-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=deburring-sticks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~5-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1791N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221791N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:05:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493325%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/material~iron-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-reducers/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/solderless-breadboards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-15795-818-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/solvent-weld/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000147754%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5790K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225790K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/duct-reducers/material~aluminum/ 2025-11-02 07:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sorbent-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=desiccants/food-industry-standard~fda-compliant-21-cfr-178-2010&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oilers/for-use-with~chlorine/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solderless-breadboards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-15795-818-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solvent-weld&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3068903585281&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=desiccants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3068903585281%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002122447%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-clamp-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25889 pages (at 47 pages/min), scraped 778 items (at 2 items/min) 2025-11-02 07:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sorbent-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:09 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=desiccants/food-industry-standard~fda-compliant-21-cfr-178-2010&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=detent-plungers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-clamps/clamp-type~pinch&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oilers/for-use-with~chlorine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~0-063/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5313593865679&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20clamps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5313593865679%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001922882%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4227901971716&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4227901971716%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965544%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pfa-pipe-nipples/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dummy-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-buckets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dial-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~7-32-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dumpers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=54285K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2254285K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9975K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229975K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~0-063inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:06:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pfa-pipe-nipples&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~0-063inches/ 2025-11-02 07:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-cutters/for-use-on~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-16995-3-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~0-21875inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9975K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229975K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-fittings/fitting-connection~barbed-2/for-use-with~ethanol/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:06:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~0-21875inches/ 2025-11-02 07:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-check-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duo-cone-seals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-grinder-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-16996-14-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-cutters/for-use-on~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16995-3-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/material~plastic-1/material~acetal/material~delrin-acetal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25937 pages (at 48 pages/min), scraped 779 items (at 1 items/min) 2025-11-02 07:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9975K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229975K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/connection-style~barbed/for-use-with~ethanol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pipe-cutters/for-use-on~aluminum/ 2025-11-02 07:07:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/material~plastic-1/material~eva/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16996-14-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964606%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-fittings/fitting-connection~barbed-2/for-use-with~phosphoric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/material~plastic/material~delrin-acetal-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/material~plastic-1/material~peek-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duo-cone-seals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dust-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plastic-sheets/material~plastic/material~delrin-acetal-plastic// 2025-11-02 07:07:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dust-collectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dust-masks/niosh-approval-number~tc-84a-0164/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/material~plastic/material~eva-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/connection-style~barbed/for-use-with~phosphoric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/material~plastic/material~peek-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plastic-sheets/material~plastic/material~eva-plastic// 2025-11-02 07:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spindle-nut-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965403%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diesel-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 07:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/fitting-connection~barbed-2/for-use-with~calcium-chloride&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 07:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plastic-sheets/material~plastic/material~peek-plastic// 2025-11-02 07:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dust-masks/niosh-approval~tc-84a-0164&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:00 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-trash-bins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-16998-45-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=digital-calipers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/dust-masks/niosh-approval~tc-84a-0164/ 2025-11-02 07:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/fitting-connection~barbed-2/for-use-with~deionized-water-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 25983 pages (at 46 pages/min), scraped 779 items (at 0 items/min) 2025-11-02 07:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spindle-nut-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushing-drivers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4657N34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224657N34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bussmann-type-atm-fuses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967804%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/fitting-connection~barbed-2/for-use-with~mineral-spirits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-trash-bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dust-mop-handles/handle-material~wood-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16998-45-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/military-specification~ms20470ad3-5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5931N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225931N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushing-drivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyimide-foam-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bussmann-type-atm-fuses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~ip20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=53505K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2253505K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dust-mop-handles/handle-material~wood&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:08:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spoilboard-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-20470-ad3-5-rivets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyimide-foam-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/dust-mop-handles/handle-material~wood/ 2025-11-02 07:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spool-stands/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=din-machine-keys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~ip20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~no-19/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-supplies/voltage~18v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26026 pages (at 43 pages/min), scraped 782 items (at 3 items/min) 2025-11-02 07:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spoilboard-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spout-oilers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~ip20/ 2025-11-02 07:09:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-21043-6-locknuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spool-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~no-40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wrenches/drive-size~t2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/buzzer-timers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~no-19&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-supplies/voltage~18v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spray-nozzle-manifolds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spout-oilers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~no-19/ 2025-11-02 07:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-21043-6-locknuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5598907428381&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5598907428381%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003552926%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~no-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spray-nozzle-misters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wrenches/size~t2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buzzer-timers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70235K736&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270235K736%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~no-40/ 2025-11-02 07:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-nozzle-manifolds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/wrenches/size~t2/ 2025-11-02 07:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cabinet-keys/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-repair-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-bellows/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-nozzle-misters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:09:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-buttons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-trolleys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26072 pages (at 46 pages/min), scraped 783 items (at 1 items/min) 2025-11-02 07:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cabinet-keys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-whip-checks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-bellows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-light-bulbs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ear-hose-clamps/pinch-clamp-style~double&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-buttons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=150l-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-protection-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6163216280169&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ear%20hose%20clamps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6163216280169%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969842%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-locating-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable-staplers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-outlet-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6541K31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226541K31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wye-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-protection-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-locating-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-staplers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hour-meters/environment~submersible&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-pipe-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-24693-c17-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7355695650549&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hour%20meters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7355695650549%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000092646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eccentric-tube-reducers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/edge-finders/manufacturer~starrett-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-precision-shim-bushing-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26121 pages (at 49 pages/min), scraped 784 items (at 1 items/min) 2025-11-02 07:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=17045T72&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2217045T72%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-24693-c48-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3%252f4%22-square-drive-ratchet-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24693-c17-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=300xl-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3%252f4%22-square-drive-ratchet-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=edge-molding&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/camera-tripods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=edge-finders/manufacturer~starrett&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/camper-jacks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24693-c48-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/square-neck-bolt-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3%252f4%22-square-drive-ratchet-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/military-specification~ms27183-44/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-safe-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/edge-finders/manufacturer~starrett/ 2025-11-02 07:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:39 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 500 Internal Server Error 2025-11-02 07:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:39 [scrapy.spidermiddlewares.httperror] (PID: 148) INFO: Ignoring response <500 https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3%252f4%22-square-drive-ratchet-wrenches&srchtxt=&mixedworld=force&workstationdata=true>: HTTP status code is not handled or not allowed 2025-11-02 07:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/squeeze-on-paint-markers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=camera-tripods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machinery-dollies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=camper-jacks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-neck-bolt-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/moving-slides/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-27183-44-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cantilever-ladders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=squeeze-on-paint-markers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dollies/high-capacity-roller-track-dollies-for-smooth-surfaces-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26162 pages (at 41 pages/min), scraped 785 items (at 1 items/min) 2025-11-02 07:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screws/military-specification~ms35206-323/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dollies/moving-slides-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-35206-230-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4236684941730&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dollies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4236684941730%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%229714204586951%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/captured-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/dollies/moving-slides-5/ 2025-11-02 07:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cantilever-ladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-buttons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/material~plastic-1/material~polystyrene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2763T38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222763T38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35206-323-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=yo-yo-harnesses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35206-230-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hydraulic-reservoirs/bowl-material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=captured-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carbon-black/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/material~plastic-1/material~acrylic-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=yor-lok-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=yor-lok-tube-tees&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carbon-packing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=yor-lok-tube-to-pipe-fittings/tube-material~stainless-steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512710%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:12:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-reservoirs/bowl-material~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/material~plastic-1/material~polyethylene-mesh&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-roller-guides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978487093033&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=yor%20lok%20tube%20to%20pipe%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978487093033%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002218529%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-black&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hydraulic-reservoirs/bowl-material~plastic/ 2025-11-02 07:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26214 pages (at 52 pages/min), scraped 786 items (at 1 items/min) 2025-11-02 07:13:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512628%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zero-flute-countersinks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-packing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4463T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224463T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/performance~easy-thermoform&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/staple-pullers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/staple-removers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elastic-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zinc-anodes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/staples-for-staplers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002466958%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-systems&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=staple-pullers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/performance~x-ray-detectable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=staple-removers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/edge-finders/manufacturer~starrett-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=staples-for-staplers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:13:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aa-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydrogen-pressure-regulating-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abs%252fpvc-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26262 pages (at 48 pages/min), scraped 787 items (at 1 items/min) 2025-11-02 07:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/performance-properties~x-ray-detectable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shoulder-screws/military-specification~ms51575-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=starrett-edge-finders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~0-005&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=double-cylinders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/plastic-sheets/performance-properties~x-ray-detectable/ 2025-11-02 07:14:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shoulder-screws/military-specification~ms51575-22/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abs-plastic-bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001455791%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shoulder-screws/military-specification~ms51575-23/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51575-16-shoulder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydrostatic-test-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cartridge-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~0-015&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:14:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=double-lip-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51575-22-shoulder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/static-control-signs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/casement-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001459678%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51575-23-shoulder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51576-23-shoulder-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/static-control-vacuums/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cartridge-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/static-control-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=static-control-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=casement-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/pipe-material~copper-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51576-23-shoulder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26309 pages (at 47 pages/min), scraped 787 items (at 0 items/min) 2025-11-02 07:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/fitting-connection~quick-disconnect&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=static-control-vacuums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ibc-funnels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=static-control-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002364704%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~1-3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/for-pipe-material~copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~10-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001466874%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/elbows/for-pipe-material~copper/ 2025-11-02 07:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51957-124-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~1-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=zip%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001631686%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3780N224&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223780N224%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/for-use-with~deionized-water-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001453224%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drain-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-124-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=zip%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001632872%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=zip%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001616157%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51957-26-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/caterpillar-grommets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~4-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/for-use-with~deionized-water&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:16:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26357 pages (at 48 pages/min), scraped 788 items (at 1 items/min) 2025-11-02 07:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001463117%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/elbows/for-use-with~deionized-water/ 2025-11-02 07:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51957-31-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/caulk-smoothing-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51957-51-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/thickness~5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-26-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caterpillar-grommets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8752K982&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228752K982%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iec-320-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastic%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001468702%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-31-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caulk-smoothing-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ignition-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-51-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms-51957-7-machine-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/fitting-connection~claw-clamp-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8702K711&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228702K711%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8752K982&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228752K982%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cellular-phone-boosters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/elbows/military-specification~ms20822-16k/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-7-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/connection-style~claw-clamp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-socket-weld-pipe-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-square-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8752K982&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228752K982%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=impact-sprinklers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26404 pages (at 47 pages/min), scraped 789 items (at 1 items/min) 2025-11-02 07:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/material~steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:17:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/adapters/connection-style~claw-clamp/ 2025-11-02 07:17:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cellular-phone-boosters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=elbows/military-specifications-met~ms20822-16k&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9980774817151&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=elbows&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9980774817151%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/step-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/centering-heads/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:24 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:17:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:17:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/elbows/military-specifications-met~ms20822-16k/ 2025-11-02 07:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/tube-od~66-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-thread-protector-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~8-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/for-use-on~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sterile-vials/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=step-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=centering-heads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=zip%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001598294%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-thumb-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/for-tube-od~66mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sticky-back-cable-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:52 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/length~8-ft&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/for-use-on~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/for-use-on~masonry/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:17:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/adapters/for-tube-od~66mm/ 2025-11-02 07:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stop-off-paint/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/for-use-on~brass/ 2025-11-02 07:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26446 pages (at 42 pages/min), scraped 789 items (at 0 items/min) 2025-11-02 07:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adapters/thread-size~5-16-24-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/overall-length~3-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indicators/for-use-with~alcohol-based-solvents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/strain-gauge-mounting-adhesives/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indicators/for-use-with~drinking-water&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-water-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9722797833693&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=zip%20ties&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9722797833693%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000121892%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stop-off-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/for-use-on~masonry/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979599325048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979599325048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179269%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979599325048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979599325048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964650%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~5-16-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:27 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zip-ties/overall-length~3-5-16&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3891N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223891N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=strain-gauge-mounting-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3515N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223515N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/adapters/thread-size~5-16-24/ 2025-11-02 07:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms3367-1-cable-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/msds-books/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 07:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 07:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastics/material~plastic-1/material~acetal/material~delrin-acetal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electric-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastics/material~plastic-1/material~acetal/material~delrin-acetal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stretchable-wire/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3891N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223891N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3515N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223515N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms3367-1-cable-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chalkboard-accessories/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=msds-books&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=778212328374&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plastics&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B778212328374%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512278%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/propylene-tanks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastics/material~plastic-1/material~plastic-1/material~acetal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26492 pages (at 46 pages/min), scraped 789 items (at 0 items/min) 2025-11-02 07:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stretchable-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3891N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223891N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chalk/for-use-on~chalkboard/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3515N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223515N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chalkboard-accessories&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/protective-clothing-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=propylene-tanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/prototype-circuit-boards/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/structural-pipe-crosses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electric-gearmotors/ul-file-number~e199928&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chalkboard-chalk&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:32 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=protective-clothing-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3535199831245&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electric%20gearmotors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3535199831245%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005015917%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastite-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=structural-pipe-crosses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=prototype-circuit-boards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/struts/strut-channel-framing-component~caster/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plate-dogs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chip-hooks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plate-levels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3535199831245&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electric%20gearmotors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3535199831245%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005015917%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26536 pages (at 44 pages/min), scraped 789 items (at 0 items/min) 2025-11-02 07:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=struts/strut-channel-framing-component~caster&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stud-welding-systems/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/muslin-fabric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electric-oilers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inserts/for-use-in~blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chip-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/formulation~silica&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2427054843101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2427054843101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143873%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3535199831245&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electric%20gearmotors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3535199831245%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005015917%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002105315%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2356T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222356T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stud-welding-systems&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=muslin-fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plates/material~aluminum-2/material~a356-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electric-heaters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/inserts/for-use-in~hollow-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/formulation~vinyl-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2356T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222356T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sump-pump-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plates/material~aluminum/material~a356-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sun-shades/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002141181%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/manufacturer~loctite/manufacturer-model-number~3201&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/manufacturer~loctite/manufacturer-model-number~5240&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9965629998276&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plates&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9965629998276%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512436%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inserts/for-use-in~hollow-block&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7649T12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227649T12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002242151%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126232%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2356T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222356T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electric-heaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002242151%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089900%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sun-shades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/inserts/for-use-in~hollow-block/ 2025-11-02 07:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sump-pump-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26586 pages (at 50 pages/min), scraped 790 items (at 1 items/min) 2025-11-02 07:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1700A27&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221700A27%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1676N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221676N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/chuck-keys/chuck-key-number~k7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inserts/thread-size~1-1-2-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ptfe-paste/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/manufacturer~loctite/manufacturer-model-number~mr-5923&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/circle-cotters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9893535896715&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=inserts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9893535896715%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334703%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-analyzers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=platform-ladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chuck-keys/key-number~k7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002242151%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002148173%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pleated-air-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plier-jaw-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ptfe-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:47 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:21:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/chuck-keys/key-number~k7/ 2025-11-02 07:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-boxes/cable-designation~sjeoow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/swage-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pull-action-toggle-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/threaded-hole-adjusters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas1149-fn232p/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/circular-charts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pull-dowels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26630 pages (at 44 pages/min), scraped 792 items (at 2 items/min) 2025-11-02 07:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aflas-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=swage-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pull-action-toggle-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjusters/mount-type~threaded-hole&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1149-fn232p-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=circular-charts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inside-hole-micrometers/manufacturer-model-number~124az&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pull-dowels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/adjusters/mount-type~threaded-hole/ 2025-11-02 07:22:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4985731609619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=inside%20hole%20micrometers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4985731609619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002087477%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clamping-shanks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clean-room-catches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nas1352c-06-12-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inside-hole-micrometers/manufacturer-model-number~823az&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clean-room-doors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plug-gauges/gauge-measurement-type~no-go/size-checking-gauge-class~z&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4985731609619&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=inside%20hole%20micrometers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4985731609619%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/cable-awg~16-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/color~black-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamping-shanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2787390765182&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plug%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2787390765182%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969294%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001466411%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clean-room-catches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1352c-06-12-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=instant-bond-adhesive-gel-for-manual-dispensing-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561177%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=23065A503&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2223065A503%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=installation-tools-for-smooth-rivet-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26680 pages (at 50 pages/min), scraped 792 items (at 0 items/min) 2025-11-02 07:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clean-room-storage-cabinets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/color~black&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-handles/drive-size~2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulated-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561177%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/plugs/color~black/ 2025-11-02 07:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=23065A503&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2223065A503%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/cable-awg~16-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/color~green-yellow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clean-room-storage-cabinets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cleaners/for-use-on~coolant/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001562847%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/color~red-yellow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-handles/size~2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002197508%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/fitting-connection~compression-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001561177%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=23065A503&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2223065A503%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002193688%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/t-handles/size~2-mm/ 2025-11-02 07:23:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pulleys/belt-trade-size~spb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cleaners/for-use-on~coolant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/fitting-connection~quick-disconnect/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-slotted-framing-hinges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6897K131&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226897K131%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/connection-style~compression&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:23:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:23:59 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9998225260164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9998225260164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929866%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/plugs/connection-style~compression/ 2025-11-02 07:24:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26727 pages (at 47 pages/min), scraped 793 items (at 1 items/min) 2025-11-02 07:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/for-belt-trade-size~spb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/idler-pulleys/belt-type~round/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/fitting-connection~quick-disconnect-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/idler-pulleys/belt-type~v/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clutch-collets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=t-slotted-framing-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pulleys/for-belt-trade-size~spb/ 2025-11-02 07:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~3-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7081K38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227081K38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/plugs/fitting-connection~quick-disconnect-1/ 2025-11-02 07:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pump-mufflers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/round-belt-idler-pulleys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498994%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/performance~interference-shielding&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/v-belt-idler-pulleys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clutch-collets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tachometers-with-nist-certificate/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pulleys/round-belt-idler-pulleys/ 2025-11-02 07:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:40 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:24:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/for-use-with~boric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insulation&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002117475%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pulleys/v-belt-idler-pulleys/ 2025-11-02 07:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tack-rags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pump-mufflers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498994%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964962%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5537K132&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225537K132%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/natural-rubber-sheet-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tachometers-with-nist-certificate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3657N131&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223657N131%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:25:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation-fill&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tack-rags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26775 pages (at 48 pages/min), scraped 794 items (at 1 items/min) 2025-11-02 07:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/cable-designation~stow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498994%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation-hanging-welders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coffee-dispensers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=natural-rubber-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pumps/power~11-64-hp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003014728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coffee-packets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coffee-urns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:27 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:25:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pumps/power~5-hp-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coffee-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pumps/power~11-64-hp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/for-use-with~plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/needle-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coffee-packets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coffee-urns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/flammability-rating~ul-94-v-0&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3556277725899&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3556277725899%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000006135051%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964595%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pumps/power~5-hp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/needles/needle-gauge~20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116718%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7827N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227827N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=needle-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pumps/power~5-hp/ 2025-11-02 07:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cold-galvanizing-paint/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cold-protection/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~1-1-16-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26819 pages (at 44 pages/min), scraped 796 items (at 2 items/min) 2025-11-02 07:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=needles/needle-gauge~20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/length~50-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collar-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cold-galvanizing-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2577530562040&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2577530562040%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079103%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cold-protection&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alloy-20-pipe-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alloy-20-stainless-steel-flat-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~1-1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001623806%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=75165A123&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2275165A123%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/plugs/material~steel/ 2025-11-02 07:26:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collar-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/punches/for-hole-diameter~1-1-16/ 2025-11-02 07:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~1-3-8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nema-10-50-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/specifications-met~ul-1309&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alumina-oxide-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum/material~aluminum-2/material~5083-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092536%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tap%252fdrill-combination-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~1-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/ul-file-number~e46194&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=intrusive-grippers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-10-50-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8248K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228248K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/punches/for-hole-diameter~1-3-8/ 2025-11-02 07:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/pipe-size~6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26868 pages (at 49 pages/min), scraped 797 items (at 1 items/min) 2025-11-02 07:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005278826%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum/material~5083-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum/material~aluminum-2/material~5052-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000047728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8046K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228046K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cable/wire-construction~stranded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510208%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/thread-type~bspp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7817035884102&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7817035884102%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003931378%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum/material~aluminum/material~5052-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-closure-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum/material~aluminum-2/material~2011-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plugs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000037076%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum/material~aluminum-2/material~6101-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510208%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511972%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510917%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ionizing-blowers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512093%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~37-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3031N2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223031N2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~43-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9397T12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229397T12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5069306377488&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5069306377488%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510208%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nema-l14-30-receptacles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-connector-boots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~5-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26917 pages (at 49 pages/min), scraped 799 items (at 2 items/min) 2025-11-02 07:28:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iron-acme-wing-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/size~37-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3031N2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223031N2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/size~43-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9397T12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229397T12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:21 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:28:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/collets/size~37-64/ 2025-11-02 07:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-l14-30-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/size~5-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/collets/size~43-64/ 2025-11-02 07:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plumbers-paste/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/connectors/nema-type~l18-20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~4-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/collets/size~5-32/ 2025-11-02 07:28:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3031N2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223031N2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9397T12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229397T12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/taps/component~stud-nut/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-ball-and-needle-roller-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/size~5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plumbers-lard-lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plumbers%27-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-l18-20-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:28:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/collets/size~5-mm/ 2025-11-02 07:28:57 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:29:00 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:29:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/punches/for-hole-diameter~4/ 2025-11-02 07:29:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=taps/component~stud-nut&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 26962 pages (at 45 pages/min), scraped 799 items (at 0 items/min) 2025-11-02 07:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-ball-and-needle-roller-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plungers/fastener-head-type~l-handle/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/technical-pens/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/telephone-wiring-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plungers/thread-size~5-8-11-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3987871406277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3987871406277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002271416%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cord-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cord-reels/extension-cord-length~30-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iso-22-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/combination-wrenches/drive-size~no-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2532A331&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222532A331%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plungers/handle-type~l-handle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9834004401048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cord%20reels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9834004401048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000126121%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=technical-pens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=telephone-wiring-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plungers/thread-size~5-8-11-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9219809646750&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=plungers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9219809646750%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001938773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7016K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227016K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:29:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/plungers/thread-size~5-8-11-1/ 2025-11-02 07:29:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=combination-wrenches/size~no-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nema-terminal-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2532A331&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222532A331%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-oxide-files&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/temperature-guns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/combination-wrenches/size~no-24/ 2025-11-02 07:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/temperature-indicating-stickers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/common-sense-buttons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7016K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227016K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ja-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-terminal-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2532A331&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222532A331%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temperature-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27012 pages (at 50 pages/min), scraped 799 items (at 0 items/min) 2025-11-02 07:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-compressors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=temperature-indicating-stickers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=common-sense-buttons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7016K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227016K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/push-pull-rods/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:30:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/neoprene-rubber-rod-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cords/color~fluorescent-pink&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tesla-meters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7010901550041&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7010901550041%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002194654%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:34 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cords/enclosure-rating~nema-3r&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-service-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-seamless-pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-pull-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-glue-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=neoprene-rubber-rod-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-sheets/material~aluminum-2/material~1235-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8208K103&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228208K103%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7010901550041&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7010901550041%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000120458%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tesla-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5481N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225481N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pans/space-saving-features~nestable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-sheets/material~aluminum-2/material~7075-aluminum/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-cords/nema-type~6-50/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/material~aluminum/material~1235-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-knockouts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27062 pages (at 50 pages/min), scraped 801 items (at 2 items/min) 2025-11-02 07:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-metallic-tubing-(emt)-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/theater-stage-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510654%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jam-nuts/thread-size~1-3-4-12-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nestable-pans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/material~aluminum/material~7075-aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cords/nema-style~6-50&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:25 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:31:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-cords/specifications-met~iec-60320&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-presses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nfpa-cylinders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509894%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=theater-stage-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/electrical-cords/nema-style~6-50/ 2025-11-02 07:31:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~1-3-4-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7010901550041&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7010901550041%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002119291%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8885K811&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228885K811%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/jam-nuts/thread-size~1-3-4-12/ 2025-11-02 07:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-shears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nfpa-cylinders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/computer-shelving/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/computer-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-solenoid-control-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-regulators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-tool-lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/putty/for-use-on~leather-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27110 pages (at 48 pages/min), scraped 802 items (at 1 items/min) 2025-11-02 07:32:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=computer-shelving&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-sockets/electrical-connection~barrel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=computer-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:32:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-vacuums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8358313672822&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20sockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8358313672822%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003411389%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~7-8-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pocket-clip-blowguns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=putty/for-use-on~leather&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thermostat-knobs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/battery-chemistry~nickel-metal-hydride-nimh/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=940728682702&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=jam%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B940728682702%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333384%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-sheets/thickness~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/putty/for-use-on~leather/ 2025-11-02 07:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pocket-hole-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/putty/for-use-on~plastic-1/for-use-on~uhmw-polyethylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:41 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~7-8-20&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9959756966201&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9959756966201%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001453294%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conductive-containers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~m27&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=thermostat-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nickel-metal-hydride-(nimh)-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=940728682702&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=jam%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B940728682702%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333390%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=putty/for-use-on~uhmw-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-wire/number-of-wires~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=janitorial-sinks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conductive-containers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvc-butterfly-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:03 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/thread-size~m27&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:33:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/putty/for-use-on~uhmw-plastic/ 2025-11-02 07:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pole-jacks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27160 pages (at 50 pages/min), scraped 802 items (at 0 items/min) 2025-11-02 07:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conduit-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4072418438324&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4072418438324%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001523609%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conduit-reducing-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/for-use-with~air/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pole-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/three-leg-bridle-slings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-butterfly-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electronics-enclosures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conduit-reducing-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jars/capacity~500-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=three-leg-bridle-slings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/for-use-with~lubricant/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=emergency-wash-stations&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tig-cups/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=62075583092&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=jars&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B62075583092%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002080009%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=enclosure-accessories&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/for-use-with~methyl-bromide-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/continuous-length-compression-springs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:33:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/contour-gauges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jars/capacity~205-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/for-use-with~silicon-tetrafluoride/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27204 pages (at 44 pages/min), scraped 802 items (at 0 items/min) 2025-11-02 07:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~methyl-bromide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=continuous-length-compression-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nipples/for-use-with~lubricants/ 2025-11-02 07:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tilt-fans/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-pigtails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=enclosures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nipples/for-use-with~methyl-bromide/ 2025-11-02 07:34:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=contour-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jars/capacity-weight-~205-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jars/capacity~300-ml/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~silicon-tetrafluoride&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:34:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/jars/capacity-weight-~205-ml/ 2025-11-02 07:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tilt-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/time-card-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tilt-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002963050%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coolant-fluid-deodorizers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/pipe-schedule~40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jars/capacity-weight-~300-ml&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=79215A73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2279215A73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ansi-40-roller-chain-attachments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tilt-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/jars/capacity-weight-~300-ml/ 2025-11-02 07:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=time-card-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:50 [scrapy.core.downloader.handlers.http11] (PID: 148) WARNING: Got data loss in https://www.mcmaster.com/products/angle-needle-valves/. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests 2025-11-02 07:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/angle-needle-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coolant-fluid-deodorizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/schedule~40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ansi-safety-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000094382%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-receptacles/connector-shell-size~12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvdf-hose-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27254 pages (at 50 pages/min), scraped 803 items (at 1 items/min) 2025-11-02 07:35:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/thread-size~3-8-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=angle-needle-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anti-static-vacuum-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polycarbonate-rectangular-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/thread-size~5-16-24-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=541000574830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20receptacles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B541000574830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001567097%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-receptacles/enclosure-rating~ip68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvdf-reducing-bushings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6134T61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226134T61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvdf-hose-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/thread-size~3-8-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~px76a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=541000574830&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20receptacles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B541000574830%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026097%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/squares/material~petg/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pyrex/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/thread-size~5-16-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2320N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222320N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvdf-reducing-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nipples/thread-size~5-16-24/ 2025-11-02 07:35:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/toilet-partition-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quad-receptacles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=px76a-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyester-(petg)-squares&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pyrex&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coolant-hose/material~buna-n/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=toilet-partition-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quarter-turn-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quad-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27299 pages (at 45 pages/min), scraped 805 items (at 2 items/min) 2025-11-02 07:36:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyester-grade-industrial-laminate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-sockets/number-of-circuits~12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nitrile-coolant-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-sockets/number-of-circuits~4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-switch-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-angle-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8358313672822&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20sockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8358313672822%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001523718%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quarter-turn-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8358313672822&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20sockets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8358313672822%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548635%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8302N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228302N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=equipment-movers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/no-1xsp-clipper-belt-lacing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ergonomic-force-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-drain-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8302N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228302N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/top-notch-lathe-tool-inserts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-drums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/torch-strikers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-end-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/penlights/material~stainless-steel-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=no-1xsp-clipper-belt-lacing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/torque-limiting-extension-bars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=top-notch-lathe-tool-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/torque-sockets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-wire/american-wire-gauge-awg~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27347 pages (at 48 pages/min), scraped 807 items (at 2 items/min) 2025-11-02 07:37:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-wire/cable-designation~sgx&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torch-strikers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4072418438324&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4072418438324%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002144939%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torches/stainless-steel-penlights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torque-limiting-extension-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4072418438324&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4072418438324%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005212328%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torque-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6176551385129&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6176551385129%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%225631449511214%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:37:30 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/torsion-cable-drums/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6070N17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226070N17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:37:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ball-screws/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2431N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222431N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7965558737631&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ball%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7965558737631%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=key-duplicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=expansion-joints/pipe-size~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=torsion-cable-drums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyolefin-pin-terminals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrical-wire/wire-finish~tin-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7692142082491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=expansion%20joints&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7692142082491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000050647%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyolefin-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=key-panel-actuators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4072418438324&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrical%20wire&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4072418438324%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002214295%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keyboard-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quill-kits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27398 pages (at 51 pages/min), scraped 809 items (at 2 items/min) 2025-11-02 07:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9175K39&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229175K39%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyolefin-u-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keyboards-and-mouse/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~1054wb/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=r395%252f25-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/battery-size~aaa-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~1001en/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quill-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nonsparking-adjustable-wrenches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electrodes/aws-material-code~e70c-6m&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~1054wb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keyboards-and-mouse&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/battery-size~aaa&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~1001en&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994383438024&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=electrodes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994383438024%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324478%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~1054wb/ 2025-11-02 07:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonsparking-adjustable-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/batteries/battery-size~aaa/ 2025-11-02 07:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~997x/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=batteries/battery-size~u1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=battery-chargers/battery-voltage~12v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~1001en/ 2025-11-02 07:38:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:38:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:38:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extractor-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tree-clips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3186695198062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=batteries&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3186695198062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003031025%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8428166447802&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=battery%20chargers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8428166447802%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001616098%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/curb-stops/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/beams/material~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27446 pages (at 48 pages/min), scraped 810 items (at 1 items/min) 2025-11-02 07:39:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extruded-aluminum-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~997x&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/radial-seals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-rope&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tree-clips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~997x/ 2025-11-02 07:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=curb-stops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=beams/material~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~cg16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/radiators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-adapter-sleeves/shaft-diameter~1-11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eye-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=radial-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/radius-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/beams/material~stainless-steel/ 2025-11-02 07:39:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:39:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4305942166524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearing%20adapter%20sleeves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4305942166524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002270122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nonwoven-abrasive-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~cg16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=radiators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:49 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearing-adapter-sleeves/shaft-diameter~1-11-16&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~k101/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cut-to-shape-tool-silhouettes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=radius-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~cg16/ 2025-11-02 07:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonwoven-abrasive-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~na6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/notching-snips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~1623&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~k101&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27489 pages (at 43 pages/min), scraped 810 items (at 0 items/min) 2025-11-02 07:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cut-to-shape-tool-silhouettes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electronic-torque-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rails/component~base/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002106277%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~k101/ 2025-11-02 07:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~ro7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~na6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=notching-snips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~s68/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-crossovers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rails/component~base&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~na6/ 2025-11-02 07:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~sc1041r/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~3201&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~ro7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-glides/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9948522430991&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9948522430991%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002235326%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~s68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fans/voltage~277v-ac-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123112%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-crossovers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~ro7/ 2025-11-02 07:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1743A16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221743A16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~s68/ 2025-11-02 07:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~sc1041r&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~sk1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=tube-glides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:40:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~sc1041r/ 2025-11-02 07:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/voltage~277v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nozzles/for-use-with~nitric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123112%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rafter-squares/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rails/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1743A16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221743A16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/fans/voltage~277v-ac/ 2025-11-02 07:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27536 pages (at 47 pages/min), scraped 810 items (at 0 items/min) 2025-11-02 07:41:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/equivalent-key-number~sk1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyurethane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/for-use-with~nitric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/equivalent-key-number~sk1/ 2025-11-02 07:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123112%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1743A16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221743A16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rafter-squares&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6308&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pop-rivets/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965017%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002087051%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=54335K64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2254335K64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:38 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pop-rivets/material~110-copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:41 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6704&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pop-rivets/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6706&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pop%20rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511329%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002090072%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pop-rivets/material~aluminum/ 2025-11-02 07:41:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=emergency-wash-shower-curtains&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002083540%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:41:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6806&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=90217A610&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2290217A610%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~0644&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/width~24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27587 pages (at 51 pages/min), scraped 811 items (at 1 items/min) 2025-11-02 07:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002135358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400414%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/bearing-trade-number~6903&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2355999507944&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fastening%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2355999507944%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000124920%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~1007la/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~0862&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=11775A807&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2211775A807%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~0892&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/twist-lock-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1618N3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221618N3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400929%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002135358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fdm-filaments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400633%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~1007la&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=11775A807&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2211775A807%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/twist-to-length-shipping-tubes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=twist-lock-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1618N3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221618N3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~1007la/ 2025-11-02 07:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tig-torch-nozzles/tig-industry-number~54n14/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002135358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feet/mount-type~rest-on/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tire-inflator-gauges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=11775A807&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2211775A807%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=twist-to-length-shipping-tubes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1618N3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221618N3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/tig-torch-nozzles/industry-number~54n14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:43:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27636 pages (at 49 pages/min), scraped 812 items (at 1 items/min) 2025-11-02 07:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=portable-demagnetizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feet/mount-type~freestanding&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/tire-inflator-gauges-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=female-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nozzles/tig-torch-nozzles/industry-number~54n14/ 2025-11-02 07:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8031429930803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=feet&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8031429930803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010917%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/diameter-tape-measures/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%22422580230907%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=portable-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/material~carbide/shank-type~round/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6113A11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226113A11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/numbered-zip-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=portable-shelters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=portable-stools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=diameter-tape-measures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=portable-walls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nurdles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/carbide-round-shank-reamers-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:43:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=porting-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/for-use-on~graphite-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=numbered-zip-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/reamers/carbide-round-shank-reamers-10/ 2025-11-02 07:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/die-grinder-collets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:43:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fh-632-10-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fhs-0420-24-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nurdles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27682 pages (at 46 pages/min), scraped 813 items (at 1 items/min) 2025-11-02 07:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fhs-632-8-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/for-use-on~graphite&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~21323&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=die-grinder-collets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fh632-10-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~3-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/reamers/for-use-on~graphite/ 2025-11-02 07:44:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fhs0420-24-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400421%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dimmer-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-caps/fitting-connection~yor-lok-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fhs632-8-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000131201%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12095A34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212095A34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6395309901074&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20caps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6395309901074%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361222%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/for-housing-id~45-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dimmer-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~25747&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-caps/material~fiberglass-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000131201%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-cutters/for-use-on~hard-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/din-rails/height~80-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:44:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:44:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401451%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6395309901074&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20caps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6395309901074%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003423545%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/nut-type~locknut/thread-size~1-3-4-5-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-0725/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5499300003000&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20cutters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5499300003000%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929991%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12095A75&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212095A75%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/for-housing-id~45mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2761N34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222761N34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27729 pages (at 47 pages/min), scraped 814 items (at 1 items/min) 2025-11-02 07:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/union-check-valves already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3738A1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223738A1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=din-rails/height~80mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000131201%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/for-housing-id~45mm/ 2025-11-02 07:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=potentiometers/ul-file-number~e68568&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~1-3-4-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0725&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/din-rails/height~80mm/ 2025-11-02 07:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dish-soap/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7458750031228&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=potentiometers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7458750031228%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005408241%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000131898%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/display-stands/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:32 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/union-check-valves already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=potting-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7506T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227506T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:38 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0725&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-0885/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fillers/for-use-on~acrylic-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dish-soap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=display-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5826054618903&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fillers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5826054618903%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002940773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~51533&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0885&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=utility-pole-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:45:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 07:45:58 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 07:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400779%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-0985/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pourable-epoxies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~74256&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000095542%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/nut-type~locknut/thread-size~m42/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27777 pages (at 48 pages/min), scraped 818 items (at 4 items/min) 2025-11-02 07:46:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-style-strainers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401028%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:14 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0885&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:46:14 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~0-028&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0985&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dock-door-shrouds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12095A122&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212095A122%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-1253/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~m42&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~a473&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=v-style-strainers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001715176%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000058788%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453388%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/for-use-with~odors-from-organic-vapors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002399093%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8832A45&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228832A45%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dock-door-shrouds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12095A122&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212095A122%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1253&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:44 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~m42&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984536%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~bes12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-1285/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~bes4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000077421%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5450T45&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225450T45%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-1300/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:46:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-1315/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8832A45&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228832A45%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=12095A122&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2212095A122%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27823 pages (at 46 pages/min), scraped 819 items (at 1 items/min) 2025-11-02 07:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~bes12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1285&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~bes4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000077421%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1300&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~bes12/ 2025-11-02 07:47:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000073998%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1315&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8832A45&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228832A45%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:22 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:47:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~bes4/ 2025-11-02 07:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-resistant-clothing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000020341%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ball-bearings/shaft-type~hex/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000025149%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/door-bell-buttons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000077421%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:38 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1300&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vacuum-sealable-jars/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:41 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1315&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:42 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/hex-bore-ball-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vacuum-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/vacuum-sealable-bottles already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-bell-buttons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:52 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:47:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-retardants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vacuum-tables/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-sealable-jars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:47:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bearings/hex-bore-ball-bearings/ 2025-11-02 07:47:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vacuum-sealable-bottles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/id~4-125/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/door-edge-mount-hinges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27868 pages (at 45 pages/min), scraped 819 items (at 0 items/min) 2025-11-02 07:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~d1054k/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vacuum-sealable-bottles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~4-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/id~5-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-cords/voltage~4-5v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~eg2r/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-edge-mount-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493072%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001496872%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~1-5-8-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4895046738192&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=power%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4895046738192%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553720%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~d1054k&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/door-lock-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~cat1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70235K26&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270235K26%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~eg2r&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~d1054k/ 2025-11-02 07:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vanadium-steel-taps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400730%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-5-8-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~eg2r/ 2025-11-02 07:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8911K373&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228911K373%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/door-mounting-hardware-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=door-lock-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1155A21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221155A21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448688%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~2-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-1640/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vanadium-steel-taps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/variable-output-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27913 pages (at 45 pages/min), scraped 821 items (at 2 items/min) 2025-11-02 07:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~20-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001732642%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bearings/od~6-1875/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1155A21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221155A21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001726455%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8940A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228940A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1640&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=variable-output-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~3-137-12-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mills/mill-diameter~3-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8940A44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228940A44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/shaft-diameter~1-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000089122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/od~6-3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9654196912961&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=end%20mills&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9654196912961%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001727615%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038829%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~3-137-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000076905%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~3-4-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/shaft-diameter~1-13-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3034A313&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223034A313%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000089122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanged-butterfly-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/nuts/thread-size~3-137-12/ 2025-11-02 07:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002260089%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~mc51&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~3-4-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/double-loop-couplings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3034A313&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223034A313%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 27964 pages (at 51 pages/min), scraped 823 items (at 2 items/min) 2025-11-02 07:50:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400432%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000089122%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanged-hydraulic-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanged-standoffs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=double-loop-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3405N321&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223405N321%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400432%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3034A313&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223034A313%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~o997e/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-supplies/certification~siq-type-approved&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/keys/key-number-equivalent~r1003m/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dow-corning-732-adhesives%252fsealants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-supplies/output-voltage~5v-dc-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dowel-pin-assortments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bearings/shaft-diameter~20-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400432%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5598907428381&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=power%20supplies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5598907428381%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002276400%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:48 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~o997e&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=935780893697&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B935780893697%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002266865%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vibration-damping-plates/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:50:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/key-number~r1003m&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3720N128&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223720N128%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~o997e/ 2025-11-02 07:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~m140/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:51:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dowel-pins/dowel-pin-assortments-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28011 pages (at 47 pages/min), scraped 824 items (at 1 items/min) 2025-11-02 07:51:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/keys/key-number~r1003m/ 2025-11-02 07:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-supplies/certification~as-international-association-certified/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/vibration-tables/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/dowel-pins/dowel-pin-assortments-5/ 2025-11-02 07:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vibration-damping-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-2245/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/welding-supplies/component~cutting-attachment/manufacturer-model-number~ca411-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m140&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flared-fittings/gender~male-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~m36/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drafting-tape-dispensers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-supplies/specifications-met~as-international-association-certified&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=vibration-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451255%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9937322946106&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flared%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9937322946106%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000019882%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flared-swivel-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2245&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/power-supplies/specifications-met~as-international-association-certified/ 2025-11-02 07:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3720N128&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223720N128%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=victor-ca411-3-oxyacetylene-cutting-attachments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000011118%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m36&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drafting-tape-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bellows/material~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:57 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2245&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~m68/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334843%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3720N128&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223720N128%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=engraving-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-washer-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28057 pages (at 46 pages/min), scraped 824 items (at 0 items/min) 2025-11-02 07:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bellows/material~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bevel-gear-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epdm-rubber-bar-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epdm-rubber-duct&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:52:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bellows/material~stainless-steel/ 2025-11-02 07:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451801%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bit-dispensers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epdm-rubber-rod-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bit-stop-collars/performance~depth-limiting/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-cup-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flexible-cable-trays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-6%252f6-threaded-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/width~1-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451801%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wall-file-pockets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bit-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/depth-limiting-drill-bit-stop-collar-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-extension-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000087555%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-chip-catchers/for-machining-type~drilling/for-use-on~metal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-cup-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:52:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:52:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/depth-limiting-drill-bit-stop-collar-sets/ 2025-11-02 07:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451801%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/color~yellow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wall-file-pockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28108 pages (at 51 pages/min), scraped 824 items (at 0 items/min) 2025-11-02 07:53:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wallpaper-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/floats/for-use-with~hydraulic-fluid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-3080/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/magnetic-chip-catchers-for-drilling-metal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=float-level-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~1-2-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/magnetic-chip-catchers-for-drilling-metal/ 2025-11-02 07:53:23 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wallpaper-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334370%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floats/for-use-with~hydraulic-fluid-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3080&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epoxies/for-use-on~plastic-1/for-use-on~uhmw-polyethylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-axial-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/floats/for-use-with~hydraulic-fluid-1/ 2025-11-02 07:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000090442%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kimwipes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-push-on-hose-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bolts/thread-size~3-8-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~10110&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:51 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3080&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/water-demineralizers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/for-use-on~uhmw-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6342366933521&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6342366933521%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454348%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-brake-benders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~11-8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/epoxies/for-use-on~uhmw-plastic/ 2025-11-02 07:54:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-push-on-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28155 pages (at 47 pages/min), scraped 824 items (at 0 items/min) 2025-11-02 07:54:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74575A81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274575A81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-control-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-spacers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-demineralizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~14277&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/water-hammer-suppressors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:54:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~11-8-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002150867%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~11-8-mm/ 2025-11-02 07:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nylon-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-spacers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-fit-nuts/material~steel-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=66215A32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2266215A32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=water-hammer-suppressors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~608&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~21-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~8267&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9946404817603&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=press%20fit%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9946404817603%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511437%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002095504%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126921%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-3715/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6430A44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226430A44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~21-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7605A2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227605A2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxies/manufacturer-model-number~e-120hp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-rate-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~21-5-mm/ 2025-11-02 07:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2997781987034&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=epoxies&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2997781987034%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002150045%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28204 pages (at 49 pages/min), scraped 828 items (at 4 items/min) 2025-11-02 07:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3715&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-switches/certification~ce-marked&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~3-9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/brackets/material~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-fit-nuts/pem-part-number~cls-348-0&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6430A24&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226430A24%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-ring-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000054258%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8779931053344&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8779931053344%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002230942%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~35-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9946404817603&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=press%20fit%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9946404817603%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108514%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:24 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3715&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:55:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:55:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~3-9-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/material~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/knobs/thread-size~5-8-11-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-ring-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ball-valves-for-natural-gas/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~35-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~3-9-mm/ 2025-11-02 07:55:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/brackets/material~fiberglass/ 2025-11-02 07:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~4-15-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-3520/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:55:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/mount-type~threaded-stud&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~35-64/ 2025-11-02 07:55:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/webbing-snap-hooks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knobs/thread-size~5-8-11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-valves/valve-type~ball/for-use-with~natural-gas&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=brackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010668%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/mounting-location~pipe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/knobs/thread-size~5-8-11/ 2025-11-02 07:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~4-15-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-valves/valve-type~ball/pipe-size~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flowmeter-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=brackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002238481%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=webbing-snap-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~4-15-mm/ 2025-11-02 07:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28252 pages (at 48 pages/min), scraped 829 items (at 1 items/min) 2025-11-02 07:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/label-printers/manufacturer~epson-k-sun-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~53-ga-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=339191585183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B339191585183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247047%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000158851%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/weld-in-pull-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brackets/screw-size~no-9-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989682661706&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=brackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989682661706%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334324%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epson-label-printers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~53-ga&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=l-key-organizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=equipment-leakage-current-interrupters-(elcis)&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-alloy-385&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=weld-in-pull-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=equipment-seats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1556A26&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221556A26%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~53-ga/ 2025-11-02 07:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-hole-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~117/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/welding-coolant-recirculators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~9-1-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~9-8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-tape-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=er-5356-welding-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=er-80s-d2-welding-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28297 pages (at 45 pages/min), scraped 830 items (at 1 items/min) 2025-11-02 07:57:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~117&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=erasers/for-use-on~chalkboard&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-coolant-recirculators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~9-1-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/size~9-8-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000011456%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6379782805883&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=erasers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6379782805883%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930289%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~154/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~9-1-mm/ 2025-11-02 07:57:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drill-bits/size~9-8-mm/ 2025-11-02 07:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-4485/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5724T14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225724T14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=folding-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-4580/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=erector-reamers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~178/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/welding-pencils/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~154&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4485&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~206/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000039790%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-4625/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-4650/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4580&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~178&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:57:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=welding-pencils&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000106721%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000130603%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28343 pages (at 46 pages/min), scraped 831 items (at 1 items/min) 2025-11-02 07:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000093899%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:11 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4485&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:58:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~206&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foot-pegs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4625&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4650&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:17 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4580&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~227/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K336&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K336%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086888%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-to-close-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000131089%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000023801%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ethernet-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~247/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/forceps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:37 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4625&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:58:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:58:38 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4650&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~227&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K336&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K336%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-4965/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000117464%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wheel-riser-ramps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~247&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=forceps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ethernet-port-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K336&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K336%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000155684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eva-foam-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28389 pages (at 46 pages/min), scraped 831 items (at 0 items/min) 2025-11-02 07:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fractional-jobbers-drills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wheel-riser-ramps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-5090/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~329/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-threaded-one-end-pipe-nipples&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 07:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~5-mm-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=brass-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/white-spirits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~344/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/maximum-pressure~1000-psi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-5090&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-control-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~329&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000040676%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wind-screens/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000143458%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=white-spirits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~344&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/maximum-pressure-psi~1000-psi-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:50 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-5090&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 07:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dusters/material~lambswool-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-fan-guards/blade-diameter~48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~5-mm/ 2025-11-02 07:59:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 07:59:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000012934%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/exhaust-fan-guards/blade-diameter~18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 07:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7534361604797&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pressure%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7534361604797%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001705615%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5195649950898&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=exhaust%20fan%20guards&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5195649950898%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000079782%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/window-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28436 pages (at 47 pages/min), scraped 831 items (at 0 items/min) 2025-11-02 08:00:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lambswool-dusters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19555K22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219555K22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~e5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=friction-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-fan-guards/for-blade-diameter~18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~373/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-alloy-220-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/exhaust-fan-guards/for-blade-diameter~18/ 2025-11-02 08:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-fans/blade-diameter~10-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19555K22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219555K22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~e5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~ep16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9102215783954&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=exhaust%20fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9102215783954%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005071654%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/window-position-brackets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~373&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laminated-shims&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~e5/ 2025-11-02 08:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1932K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221932K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000060340%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19555K22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219555K22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuel-transfer-barrel-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~381/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~ep16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/maximum-pressure~300-psi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=window-position-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K397&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K397%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exhaust-fans/blade-diameter~15-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~ep16/ 2025-11-02 08:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drive-sockets/drive-size~ip40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:01:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28485 pages (at 49 pages/min), scraped 833 items (at 2 items/min) 2025-11-02 08:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~381&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9102215783954&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=exhaust%20fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9102215783954%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005107605%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-6870/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/winter-grade-air-tool-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/maximum-pressure-psi~300-psi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=full-dog-point-set-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000125118%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drive-sockets/size~ip40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2232K36&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222232K36%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7534361604797&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pressure%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7534361604797%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001731667%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bronze-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lamp-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K437&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K437%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/drive-sockets/size~ip40/ 2025-11-02 08:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-6870&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=winter-grade-air-tool-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire/diameter~1-65-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/reamer-diameter~0-8745/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2232K36&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222232K36%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000090226%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~453/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:53 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-6870&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/dash-number~472/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K437&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K437%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/measurement-unit~newtons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire/diameter~1-65mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lapping-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-8745&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity~10-gal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2232K36&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222232K36%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~453&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/wire/diameter~1-65mm/ 2025-11-02 08:02:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000022822%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28534 pages (at 49 pages/min), scraped 833 items (at 0 items/min) 2025-11-02 08:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=104738336527&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=buckets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B104738336527%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002078531%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000166951%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~472&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K437&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K437%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/scale~newtons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:16 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-8745&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:02:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drum-pallets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4177T23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224177T23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K447&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K447%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000161743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pressure-gauges/scale~newtons/ 2025-11-02 08:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laser-beam-converters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-shears/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K466&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K466%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/for-use-with~elemental-fluorines/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drum-pallets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=buckets/capacity~35-qt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/blade-shape~straight/size-adjustability~adjustable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-relief-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=104738336527&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=buckets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B104738336527%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002086688%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wire-shears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/hardness~durometer-55d/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-test-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/for-use-with~elemental-fluorines&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wiring-trough-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/straight-blade-adjustable-size-reamers-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/hardness~durometer-75a/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28581 pages (at 47 pages/min), scraped 836 items (at 3 items/min) 2025-11-02 08:03:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dryer-vent-brushes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=104738336527&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=buckets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B104738336527%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002086688%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dryer-vents/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/hardness~durometer-55d&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/reamers/straight-blade-adjustable-size-reamers-8/ 2025-11-02 08:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drywall-taping-knives/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dual-swivel-casters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002326539%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/recessed-handles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/hardness~durometer-75a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dryer-vent-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=104738336527&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=buckets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B104738336527%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002086688%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dryer-vents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drywall-taping-knives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000155424%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=exposed-conveyors-and-moving-parts-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/system-of-measurement~metric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dual-swivel-casters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=recessed-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct/component~intake-hood/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:47 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct-booster-fans/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/workbench-tops/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extendable-scrapers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/system-of-measurement~metric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/occupancy-sensors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-style~hex-with-pilot-recess/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gasket-sealants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/component~intake-hood&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28625 pages (at 44 pages/min), scraped 836 items (at 0 items/min) 2025-11-02 08:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lathes/cut-type~internal-profiling&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001498488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-booster-fans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7160092992292&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7160092992292%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243977%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2919599794209&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lathes&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2919599794209%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243198%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct-silencers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=occupancy-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extended-tip-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:24 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct/component~intake-hood&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/office-chairs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=primers/for-use-on~brass-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wraparound-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/recycled-composites/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7559498251638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=primers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7559498251638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930266%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dunnage-shelving/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=duct-silencers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/cable-designation~cmg&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/recycled-metal/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=office-chairs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~1-100&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=wraparound-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003015571%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=recycled-composites&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107090%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/recycling-stations/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dunnage-shelving&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=recycled-metal&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2633K6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222633K6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28672 pages (at 47 pages/min), scraped 836 items (at 0 items/min) 2025-11-02 08:05:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extension-cords/electrical-connection~fiber-optic/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/cable-designation~sjoow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/for-use-on~ball-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:05:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reduced-wall-flexible-conduit/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gate-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003015458%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/dynamometers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=recycling-stations&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/fitting-connection~compression-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~21-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/connection-type~fiber-optic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5776K35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225776K35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/for-use-on~ball-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reduced-wall-flexible-conduit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/extension-cords/connection-type~fiber-optic/ 2025-11-02 08:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dynamometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929731%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/connection-style~compression&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~21mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=primers/manufacturer~dow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~3-1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=primers/manufacturer-model-number~611&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:05:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/reducers/connection-style~compression/ 2025-11-02 08:05:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bushings/for-shaft-diameter~21mm/ 2025-11-02 08:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7559498251638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=primers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7559498251638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243996%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/earplug-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=primers/manufacturer-model-number~7090&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-diameter~3-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/for-use-with~ammonia/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7559498251638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=primers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7559498251638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116932%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28717 pages (at 45 pages/min), scraped 838 items (at 2 items/min) 2025-11-02 08:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7442A111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227442A111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7559498251638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=primers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7559498251638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079278%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/gauge-size~0-018&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~3-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~6045601-002&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=earplug-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000111494%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bushings/for-shaft-diameter~3-1-4/ 2025-11-02 08:06:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-diameter~3-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/manufacturer~trico/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~anhydrous-ammonia&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/sae-viscosity-grade~10w-40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126017%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/sae-viscosity-grade~40/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19575A136&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219575A136%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bushings/for-shaft-diameter~3-32/ 2025-11-02 08:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/reducers/for-use-with~anhydrous-ammonia/ 2025-11-02 08:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/bushings/shaft-type~threaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/gauge-size~0-106&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/manufacturer~trico&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/sae-viscosity-grade~10w-40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/sae-viscosity-grade~40&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000111142%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/gauge-size~0-114&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002251132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=procedure-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003415244%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/shaft-diameter~1-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003408429%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/for-shaft-type~threaded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19575A221&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219575A221%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~a-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000132275%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1455K25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221455K25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2158K55&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222158K55%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000137366%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28768 pages (at 51 pages/min), scraped 840 items (at 2 items/min) 2025-11-02 08:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2158K37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222158K37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/bushings/for-shaft-type~threaded/ 2025-11-02 08:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002085912%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19575A229&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219575A229%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:22 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~a-6&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:07:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=procunier-2-e-tap-collets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/pipe-size~3-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/shaft-diameter~1-3-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/trade-name~lb-8008-c5-a-copper-based-anti-seize/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=procunier-2-e-tapping-attachments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/trade-name~lb-c-200-high-temperature-solid-film/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=procunier-4-f-tap-collets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=procunier-tapping-attachments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002245836%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~f-5-at&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/pipe-size~3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~lb-8008-c5-a-copper-based-anti-seize&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002130717%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~lb-c-200-high-temperature-solid-film&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/length~49-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:07:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119922%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002134068%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/shaft-diameter~3-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002093041%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001624986%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/gauge-size~80-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28817 pages (at 49 pages/min), scraped 845 items (at 5 items/min) 2025-11-02 08:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=propane-safety-gauges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~p6000-ecr-ws-rk-cs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002130717%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002241076%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:16 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:08:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393476033006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauge%20blocks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393476033006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000169120%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/trade-name~mobilgear-600-xp-680/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-crosses/fitting-connection~yor-lok-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002149483%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electrical-insulating-sockets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2249A48&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222249A48%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extension-cords/length~98-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:32 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~p6000-ecr-ws-rk-cs&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002130717%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=propane&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~mobilgear-600-xp-680&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=yor-lok-tube-crosses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/serial-port-interface~null-modem&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002090520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cords/length~98-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/shaft-diameter~made-to-order&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=bushings/thread-size~m110&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/zipper-tab-zippers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8543165603027&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extension%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8543165603027%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003015926%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2158K87&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222158K87%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:08:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/extension-cords/length~98-ft-/ 2025-11-02 08:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/electricians-tool-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002264033%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8821491978506&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8821491978506%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454097%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~s02-045&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducible-force-gas-springs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28864 pages (at 47 pages/min), scraped 847 items (at 2 items/min) 2025-11-02 08:09:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=zipper-tab-zippers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002128538%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=electricians%27-tool-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-ladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducible-force-gas-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=protective-edging&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gear-and-bearing-pullers/thread-size~m14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-equipment/manufacturer-model-number~s65-230&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/cable-designation~sjt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/trade-name~thk-afj/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reflective-vehicle-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2699428871106&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gear%20and%20bearing%20pullers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2699428871106%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1731192636428&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lavatory%20equipment&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1731192636428%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138129%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=general-purpose-mixers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=prototyping-materials&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:51 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gear-and-bearing-pullers/thread-size~m14&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/industry-designation~sjt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=external-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lawn-sprinkler-timers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:09:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~thk-afj&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reflective-vehicle-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cable/outer-jacket-color~clear/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cable/industry-designation~sjt/ 2025-11-02 08:10:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28915 pages (at 51 pages/min), scraped 847 items (at 0 items/min) 2025-11-02 08:10:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002127308%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=external-retaining-rings/retaining-ring-style~push-on&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ptfe-(schedule-80)-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass-and-ceramic-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil-filled-uhmw-polyethylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2307N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222307N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/closers/manufacturer~lcn/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/outer-insulation-color~clear&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6646950986674&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=external%20retaining%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6646950986674%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002055835%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glides/mount-type~threaded-hole-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ptfe-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cable/outer-insulation-color~clear/ 2025-11-02 08:10:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98430A114&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298430A114%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/regulators/for-use-with~chemicals-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9952801634820&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glides&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9952801634820%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010915%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-filled-uhmw-polyethylene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lcn-closers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable/voltage~16v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:44 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glides/mount-type~threaded-hole-1&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/for-use-on~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/end-mill-kits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=regulators/for-use-with~chemicals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cable-holders/mount-type~press-fit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9776671492613&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9776671492613%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553606%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extrusion-nuts/material~steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:10:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/regulators/for-use-with~chemicals/ 2025-11-02 08:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9224640831237&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable%20holders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9224640831237%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70235K51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270235K51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/for-use-on~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~rtv116&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue-dots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 28964 pages (at 49 pages/min), scraped 849 items (at 2 items/min) 2025-11-02 08:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-screw-nut-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7302661617394&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=extrusion%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7302661617394%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=end-mill-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/glue/for-use-on~fiberglass/ 2025-11-02 08:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002110661%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eye-straps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9224640831237&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable%20holders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9224640831237%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/envelopes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/regulators/for-use-with~medical-gas-blend-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil-skimmer-belts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6604A11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226604A11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epoxy-primers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/epoxy-thinners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/equipment-starts-automatically-signs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ptfe-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9224640831237&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cable%20holders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9224640831237%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=regulators/for-use-with~medical-gas-mixtures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=envelopes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxy-primers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gravity-conveyor-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=epoxy-thinners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/regulators/for-use-with~medical-gas-mixtures/ 2025-11-02 08:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=equipment-starts-automatically-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:11:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ergonomic-stools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nut-eyebolts/application~for-lifting/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyebolts/body-style~without-shoulder&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/black-oxide-eyebolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29010 pages (at 46 pages/min), scraped 851 items (at 2 items/min) 2025-11-02 08:12:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/regulators/outlet-pipe-size~1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/etchers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/calipers/distance-measured~0-mm-to-300-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1443843511411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=eyebolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1443843511411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001923991%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ethanol/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ergonomic-stools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyebolts/eyebolts-with-nuts-for-lifting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyebolts/finish~black-oxide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=regulators/outlet-pipe-size~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=etchers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pull-handles/material~zinc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=calipers/maximum-measurement~300-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/eyebolts/eyebolts-with-nuts-for-lifting/ 2025-11-02 08:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-screws/thread-size~1-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1443843511411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=eyebolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1443843511411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550737%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7367171817318&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7367171817318%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002065934%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ethanol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ethernet-receptacles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1393964000724&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pull%20handles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1393964000724%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512333%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/calipers/maximum-measurement~300-mm/ 2025-11-02 08:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1206504422411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lead%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1206504422411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cams/fitting-connection~threaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ethylene-glycol/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ok-stamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99030A033&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299030A033%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/excelprene-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:12:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cams/connection-style~threaded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-screws/thread-size~m2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ethylene-glycol&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29056 pages (at 46 pages/min), scraped 851 items (at 0 items/min) 2025-11-02 08:13:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-screws/thread-size~m28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ok-stamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caps/fitting-type~reducing-adapter&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99030A033&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299030A033%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/cams/connection-style~threaded/ 2025-11-02 08:13:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1206504422411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lead%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1206504422411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334645%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=excelprene-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1206504422411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lead%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1206504422411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454155%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pull-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7397459384976&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=caps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7397459384976%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002466710%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/expansion-chambers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97783A110&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297783A110%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7549K83&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227549K83%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99030A033&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299030A033%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/open-thrust-ball-bearings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyebolts/thread-size~m30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extendable-scrapers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=caps/pipe-size~4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/extension-cord-crossovers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=expansion-chambers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97783A110&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297783A110%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1443843511411&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=eyebolts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1443843511411%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334725%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7397459384976&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=caps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7397459384976%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129622%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=open-thrust-ball-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:13:56 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyebolts/thread-size~m30&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=extension-cord-crossovers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulley-and-sprocket-bushings/shaft-diameter~30-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97783A110&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297783A110%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grinding-wheels/grit~24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29106 pages (at 50 pages/min), scraped 852 items (at 1 items/min) 2025-11-02 08:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1185587692173&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pulley%20and%20sprocket%20bushings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1185587692173%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002257887%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=eyeglass-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=carbon-steel-hollow-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2344K34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222344K34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:14:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/grit~24-grit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:24 [scrapy.core.downloader.handlers.http11] (PID: 148) WARNING: Got data loss in https://www.mcmaster.com/organic-vapors-pumps. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests 2025-11-02 08:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/organic-vapors-pumps already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grip-lock-and-twist-claw-hose-to-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/belt-trade-size~xl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grinding-wheels/grit~24-grit/ 2025-11-02 08:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/outlets/current-voltage~1-5-amp-12v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/outlets/current-voltage~1-amp-5v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=left-hand-threaded-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8470423702633&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pulleys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8470423702633%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001999635%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=casters/wheel-diameter~2-1-2-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fall-arrest-snaps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pulleys/belt-trade-size~l/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1981438481801&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=casters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1981438481801%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001928828%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/current~1-5-a-12-v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/current~1-a-5-v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fan-propellers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/outlets/current~1-5-a-12-v-dc/ 2025-11-02 08:14:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fall-arrest-snaps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/outlets/current~1-a-5-v-dc/ 2025-11-02 08:14:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/for-belt-trade-size~l&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fanny-packs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pulleys/v-belt-type~cogged/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lego-style-transfers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pulleys/for-belt-trade-size~l/ 2025-11-02 08:15:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pumps/for-use-with~organic-vapors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fan-propellers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29152 pages (at 46 pages/min), scraped 853 items (at 1 items/min) 2025-11-02 08:15:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fall-arrest-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/timing-belt-pulleys/belt-trade-size~l/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fanny-packs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/for-v-belt-type~cogged&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cellulose-sheet-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=organic-vapors-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pulleys/for-v-belt-type~cogged/ 2025-11-02 08:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/faucet-aspirators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/material~iron-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulleys/l-series-timing-belt-pulleys-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grounding-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fence-post-pullers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8470423702633&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pulleys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8470423702633%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000147754%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8470423702633&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pulleys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8470423702633%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%222472617029576%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulling-and-lifting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fabric-fastener-installation-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=faucet-aspirators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fan-blades/number-of-blades~4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6495K16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226495K16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pump-guards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulling-and-lifting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oven-trays/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8197494541975&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fan%20blades&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8197494541975%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001518834%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fence-post-pullers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8470423702633&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pulleys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8470423702633%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000147754%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulling-and-lifting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=17545K31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2217545K31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29199 pages (at 47 pages/min), scraped 854 items (at 1 items/min) 2025-11-02 08:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ferraz-shawmut-type-gdg-fuses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:08 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 500 Internal Server Error 2025-11-02 08:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:08 [scrapy.spidermiddlewares.httperror] (PID: 148) INFO: Ignoring response <500 https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pulling-and-lifting&srchtxt=&mixedworld=force&workstationdata=true>: HTTP status code is not handled or not allowed 2025-11-02 08:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levels/thread-size~1-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oven-trays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8470423702633&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pulleys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8470423702633%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000147754%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hairpin-cotter-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pump-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4945834745524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=levels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4945834745524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454032%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/overhead-light-lenses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=17545K31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2217545K31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferraz-shawmut-type-gdg-fuses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=halogen-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/respirators/manufacturer-model-number~n75004l/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/overpack-drums/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pumps/power~1-4-hp-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overhead-light-lenses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fiberglass-packing-insulation/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oxygen-meters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/retainer-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=17545K31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2217545K31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hammer-tips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/manufacturer-model-number~n75004l&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overpack-drums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002074102%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pumps/horsepower~1-4-hp-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-packing-insulation&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29245 pages (at 46 pages/min), scraped 854 items (at 0 items/min) 2025-11-02 08:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oxygen-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retainer-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=54445T239&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2254445T239%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3556277725899&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pumps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3556277725899%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002995016%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/retaining-ring-applicators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/environment~submersible&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/file-handles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pa-system-horns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000092646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/retractable-cord-reels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pumps/voltage~115v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/packing-foam-pads/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7311N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227311N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retaining-ring-applicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chuck-keys/chuck-key-number~k32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=file-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pa-system-horns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=chuck-keys/chuck-key-number~s2a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retractable-cord-reels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pumps/voltage~115v-ac-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9311230464378&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chuck%20keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9311230464378%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000126575%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=packing-foam-pads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~1-11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9311230464378&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=chuck%20keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9311230464378%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002065019%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:17:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pumps/voltage~115v-ac-1/ 2025-11-02 08:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fire-extinguisher-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3169A23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223169A23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3169A54&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223169A54%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/padlock-control-centers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/padlock-eyes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29290 pages (at 45 pages/min), scraped 857 items (at 3 items/min) 2025-11-02 08:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~20-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~1-11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~13-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/padlock-latches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000034050%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~22&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/punches/for-hole-diameter~1-11-16/ 2025-11-02 08:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=padlock-control-centers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~17-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000027511%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=padlock-eyes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1932K33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221932K33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fiberglass-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000097098%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hanging-wire-guides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=padlock-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1963K131&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221963K131%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1907K13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221907K13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~17mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1932K33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221932K33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fire-sprinkler-head-covers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clamps/material~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=class-8-steel-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/punches/for-hole-diameter~17mm/ 2025-11-02 08:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fire-sprinkler-heads/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1963K131&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221963K131%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:18:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/clamps/material~fiberglass/ 2025-11-02 08:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~37-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clay-pipe-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1932K33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221932K33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29337 pages (at 47 pages/min), scraped 859 items (at 2 items/min) 2025-11-02 08:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-sprinkler-head-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/punches/hole-diameter~48-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-base-number~e10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-sprinkler-heads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1963K131&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221963K131%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000054944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~37mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clear-vinyl-rolls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:25 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pads/mount-type~press-fit/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/for-hole-diameter~48mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K355&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K355%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/punches/for-hole-diameter~37mm/ 2025-11-02 08:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:37 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/punches/for-hole-diameter~48mm/ 2025-11-02 08:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-blankets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-threaders/manufacturer-model-number~44540/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clips/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clevis-pins/diameter~7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pads/mount-type~press-fit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/overall-height~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flag-handle-keys/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1255618309250&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=clevis%20pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1255618309250%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000149734%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:19:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951811382783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951811382783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-resistant-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ridgid-36277-reamer-cones&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000020305%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clips/material~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fans/voltage~12v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:06 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clevis-pins/diameter~7-8&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29386 pages (at 49 pages/min), scraped 860 items (at 1 items/min) 2025-11-02 08:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-shrink-tubing/color~clear&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19115K319&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219115K319%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flag-handle-keys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/clips/material~brass/ 2025-11-02 08:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5392596535200&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fans&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5392596535200%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553683%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rigging-chain/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clips/specifications-met~asme-standard/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8375494534240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=heat%20shrink%20tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8375494534240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001527245%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-shrink-tubing/color~red&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-shape-number~edx17&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flange-mount-immersion-heater-temperature-switches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/right-angle-gear-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19115K319&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219115K319%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-shape-number~g16-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8375494534240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=heat%20shrink%20tubing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8375494534240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522000%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=clips/specifications-met~asme&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rigging-chain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002061377%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7496K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227496K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:45 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/clips/specifications-met~asme/ 2025-11-02 08:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002065253%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8293K7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228293K7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flange-mount-immersion-heater-temperature-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=19115K319&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2219115K319%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-shape-number~g25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T089&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T089%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-shrinks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002065787%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cmm-plates/length~24/length~2-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8293K7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228293K7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:07 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29440 pages (at 54 pages/min), scraped 862 items (at 2 items/min) 2025-11-02 08:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-sinks/thread-type~npt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9998759080309&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cmm%20plates&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9998759080309%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001629723%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001613476%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-shape-number~s14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1796N17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221796N17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=heat-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9615546959836&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=heat%20sinks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9615546959836%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:20 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cmm-plates/length~24/length~2-ft&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:21:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002051943%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8293K7&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228293K7%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/drive-style~drilled-spanner-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/hole-diameter~4-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=punches/hole-diameter~38-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8715K321&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228715K321%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1796N17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221796N17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000044364%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5113201929937&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5113201929937%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002244274%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5113201929937&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5113201929937%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002271947%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9615546959836&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=heat%20sinks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9615546959836%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-shape-number~t4-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:21:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/drive-style~square&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flashlight-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002066255%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1796N17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221796N17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coatings/for-use-on~sbr/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/signs/message~ring-bell-for-service/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5113201929937&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5113201929937%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002271947%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000020582%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9615546959836&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=heat%20sinks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9615546959836%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000142617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T807&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T807%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~1076&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29492 pages (at 52 pages/min), scraped 863 items (at 1 items/min) 2025-11-02 08:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flashlight-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coatings/for-use-on~titanium-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324061%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ring-bell-for-service-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5113201929937&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5113201929937%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002271947%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~sbr-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paint/for-use-on~laminate/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flaw-detection-spray/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/coatings/for-use-on~sbr-rubber/ 2025-11-02 08:22:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ring-heaters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paint/for-use-on~plastic-1/for-use-on~polypropylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=coatings/for-use-on~titanium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paint/for-use-on~plastic-1/for-use-on~pvdf/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324061%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:22:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paint/for-use-on~vinyl-fabric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint/for-use-on~laminate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/coatings/for-use-on~titanium/ 2025-11-02 08:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flexible-goosenecks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flaw-detection-spray&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ring-heaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint/for-use-on~polypropylene-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4999700653188&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4999700653188%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930310%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint/for-use-on~pvdf-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324061%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint/for-use-on~vinyl-fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:22:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/paint/for-use-on~polypropylene-plastic/ 2025-11-02 08:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/paint/for-use-on~pvdf-plastic/ 2025-11-02 08:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~1228&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/collets/collet-size~3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rinse-tanks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flexible-goosenecks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29538 pages (at 46 pages/min), scraped 864 items (at 1 items/min) 2025-11-02 08:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4999700653188&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4999700653188%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930411%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4999700653188&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4999700653188%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930310%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002320737%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:12 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:23:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/collet-size~5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T666&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T666%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helical-insert-installation-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=collets/collet-size~3mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rinse-tanks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3966958597415&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=collets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3966958597415%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003347858%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4999700653188&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4999700653188%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930310%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helical-sti-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/collets/collet-size~3mm/ 2025-11-02 08:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/finish~ultra-corrosion-resistant-coated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3257A17&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223257A17%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~1450&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/military-specification~ms122091/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivet-setters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/decals/message~push/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001540475%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321562%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:47 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/riveters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-buttons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=90122A313&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2290122A313%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6570T3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226570T3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=helicoils/military-specifications-met~ms122091&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivet-setters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-decals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-in-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/helicoils/military-specifications-met~ms122091/ 2025-11-02 08:24:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29591 pages (at 53 pages/min), scraped 868 items (at 4 items/min) 2025-11-02 08:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=riveters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~168&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paint-poles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/floor-heaters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002326403%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/diameter~0-197/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~1823&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~193&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/material~rubber-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K83&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K83%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint-poles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-heaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002327056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/material~stainless-steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002320963%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=concentration-meters/for-testing~lubricant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000164949%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/diameter~0-197&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T703&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T703%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T506&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T506%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6380903578127&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=concentration%20meters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6380903578127%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002845035%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000047366%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flow-control-valve-seats/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paired-magnets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:24:57 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=concentration-meters/for-testing~lubricant&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/material~steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98919A027&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298919A027%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fluidization-nozzles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29638 pages (at 47 pages/min), scraped 871 items (at 3 items/min) 2025-11-02 08:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-control-valve-seats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paired-magnets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cones/material~paper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-heat-paint&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-to-close-latches/key-number~ch751&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~00-90&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fluidization-nozzles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9975444946387&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cones&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9975444946387%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511418%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5522819982500&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=push%20to%20close%20latches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5522819982500%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400803%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flush-valve-handles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~28psb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451234%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=40785T5&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2240785T5%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:37 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-to-close-latches/key-number~ch751&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323678%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~3021b&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=connectors/thread-type~nptf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flush-valve-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8034K5&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228034K5%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323980%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pan-head-thread-cutting-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-to-connect-plug-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~1-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-to-connect-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=push-to-connect-tube-fittings-for-flame-retardant-polyethylene-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/panel-actuators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29690 pages (at 52 pages/min), scraped 874 items (at 3 items/min) 2025-11-02 08:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323980%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pan-head-thread-cutting-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/foam-peanuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99030A033&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299030A033%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~3175g&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/foam-swabs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/panel-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panel-actuators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323980%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321649%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-peanuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~378&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T194&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T194%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-swabs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-belts/material~buna-n&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panel-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinge-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:47 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:26:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3496939935355&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3496939935355%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~4157ll&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/food-industry-batch-cans/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hinges/mount-type~weld-on/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T128&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T128%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/putty/for-use-on~plastic-1/for-use-on~polypropylene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321665%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/robot-attachments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29736 pages (at 46 pages/min), scraped 876 items (at 2 items/min) 2025-11-02 08:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-rollers/between-frame-width~48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3496939935355&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3496939935355%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=conveyor-rollers/roller-diameter~1-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T551&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T551%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-industry-batch-cans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinges/hinges-without-holes-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/panels/material~bronze-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1360744530601&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1360744530601%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002279101%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=putty/for-use-on~polypropylene-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1360744530601&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1360744530601%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000145040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-attachments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%223186800911374%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/putty/for-use-on~polypropylene-plastic/ 2025-11-02 08:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3496939935355&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20belts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3496939935355%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/copper/material~copper-1/material~110-copper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panels/material~bronze&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1360744530601&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1360744530601%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002279101%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper/temper-rating~soft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/panels/material~bronze/ 2025-11-02 08:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%223186800911374%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~6s6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paper/abrasive-material~crocus/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-backed-strip-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~9-64-41-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5368696224678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=copper&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5368696224678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002292944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper/material~copper/material~110-copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1360744530601&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=conveyor%20rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1360744530601%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002279101%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002326040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paper/abrasive-material~garnet/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449205%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5368696224678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=copper&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5368696224678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511329%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5175K221&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225175K221%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8620K15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228620K15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29790 pages (at 54 pages/min), scraped 878 items (at 2 items/min) 2025-11-02 08:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/foundry-metals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paper/abrasive-material~crocus&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2694067320625&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hinges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2694067320625%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%223186800911374%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6350K689&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226350K689%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paper/abrasive-material~garnet&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992232473973&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992232473973%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001535371%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5175K221&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225175K221%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~9004ll&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992232473973&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992232473973%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550502%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foundry-metals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4672A65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224672A65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002325199%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~9005&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~m240&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4693A18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224693A18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5175K221&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225175K221%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing/multipurpose-flame-retardant-garolite-g-10-fr4-sheets-and-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002324316%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449402%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4672A65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224672A65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971113029698&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971113029698%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%227285722698868%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper-wire-sleeving&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002325199%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:28:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9613T32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229613T32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=copper-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:29:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/struts/strut-channel-framing-component~shelf-bracket/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=85345K112&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2285345K112%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/t-slotted-framing-hinges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29841 pages (at 51 pages/min), scraped 881 items (at 3 items/min) 2025-11-02 08:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4672A65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224672A65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rulers/material~paper/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002325199%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9613T32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229613T32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing/strut-channel-shelf-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=85345K112&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2285345K112%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hook-and-loop-discs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing/t-slotted-framing-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:25 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fasteners/thread-size~m75&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:29 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/framing/strut-channel-shelf-brackets/ 2025-11-02 08:29:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paper-rulers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:33 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:29:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/framing/t-slotted-framing-hinges/ 2025-11-02 08:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rod-brackets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9613T32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229613T32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6136281167969&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fasteners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6136281167969%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449101%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=85345K112&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2285345K112%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/parallel-ground-screwdrivers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hoppers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/color~amber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvc-pipe-tees/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:29:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/adhesive-type~rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-rectangular-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2355999507944&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fastening%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2355999507944%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548139%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rod-end-couplers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2355999507944&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fastening%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2355999507944%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001534500%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parallel-ground-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29892 pages (at 51 pages/min), scraped 881 items (at 0 items/min) 2025-11-02 08:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7648A73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227648A73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=corrugated-sheets/width~6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-end-couplers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pvc-pipe-tees/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~ehj&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/parker-compressed-air-regulators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8518130713417&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=corrugated%20sheets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8518130713417%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000157313%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/for-use-with~beverage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~elc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=20585T104&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2220585T104%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-solid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967788%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321279%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~evw&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parker-compressed-air-regulators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:42 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 08:30:42 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 08:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1535K37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221535K37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321493%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~eyb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/particleboard-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fuse-assortments/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1535K57&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221535K57%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321257%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fastening-tape/width~0-875/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:30:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~f30t12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pvc-tubes-for-chemicals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/g-force-indicators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1535K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221535K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29941 pages (at 49 pages/min), scraped 885 items (at 4 items/min) 2025-11-02 08:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323488%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334632%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=particleboard-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/parts-washer-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuse-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fastening-tape/width~7-8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1501K71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221501K71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=g-force-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2355999507944&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fastening%20tape&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2355999507944%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009487%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=countersunk-head-wood-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-045/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parts-washer-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/shaft-diameter~60-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=qa-stamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/shaft-diameter~3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-duct&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-045&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=faucets/for-manufacturer~american-standard&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings-for-nitrogen/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-shaft-diameter~60mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=faucets/for-manufacturer~delta&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-shaft-diameter~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=faucets/for-manufacturer~elkay&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542546%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503989650348&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=faucets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503989650348%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002113454%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/couplings/for-shaft-diameter~60mm/ 2025-11-02 08:32:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=faucets/for-manufacturer~t-s-brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503989650348&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=faucets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503989650348%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123171%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/couplings/for-shaft-diameter~3/ 2025-11-02 08:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 29992 pages (at 51 pages/min), scraped 887 items (at 2 items/min) 2025-11-02 08:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503989650348&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=faucets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503989650348%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002091773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~jcd&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~nitrogen&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/couplings/for-use-with~sodium-carbonate-soda-ash-20-concentration-in-water/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503989650348&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=faucets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503989650348%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002400311%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-134/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2633K66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222633K66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542546%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002323951%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/couplings/for-use-with~nitrogen/ 2025-11-02 08:32:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503989650348&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=faucets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503989650348%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123171%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9992T14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229992T14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/fitting-connection~barbed-2/thread-type~bspt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-146/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/for-use-with~sodium-carbonate-20-in-water-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-134&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/peek-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542546%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000140314%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/couplings/for-use-with~sodium-carbonate-20-in-water-/ 2025-11-02 08:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/bulb-trade-number~xpr18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503989650348&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=faucets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503989650348%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002123171%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000039858%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1003ll/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-146&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002320992%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=peek-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:32:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:32:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/shaft-diameter~1-9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548762%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K591&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K591%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:33:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-204/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30042 pages (at 50 pages/min), scraped 889 items (at 2 items/min) 2025-11-02 08:33:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002279353%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feedback-cable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1003ll&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-2188/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8893K155&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228893K155%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-disconnect-barbed-tube-fittings/material~brass-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feeler-gauges/thicknesses-included~0-001-amber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6408K18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226408K18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~1003ll/ 2025-11-02 08:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9727148641533&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=quick%20disconnect%20barbed%20tube%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9727148641533%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070378%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pegboard-cabinets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-204&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-006/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~1157ll/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-2188&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522426%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6408K18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226408K18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001523852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-0202/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pegboard-cabinets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-0060&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~1157ll&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3023A225&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223023A225%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A216&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A216%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gauge-blocks/size~0-0060/ 2025-11-02 08:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6408K18&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226408K18%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:33:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~1157ll/ 2025-11-02 08:33:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-0202&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-339/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-disconnect-swivel-hose-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~0-227/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gauge-blocks/size~0-0202/ 2025-11-02 08:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30086 pages (at 44 pages/min), scraped 893 items (at 4 items/min) 2025-11-02 08:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-exhaust-relief-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=couplings/thread-size~4-48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-339&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cpvc-pipe-nipples&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981342947220&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=couplings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981342947220%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334791%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~1-11-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cpvc-square-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001532761%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-227&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-wrap&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-to-pipe-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A201&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A201%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001551518%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/drive-size~1-11-16-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/crowfeet/drive-size~1-9-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~0-034&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/crowfeet/drive-size~1-11-16-1/ 2025-11-02 08:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8893K195&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228893K195%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quiet-stem-casters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/feeler-gauge-thicknesses-included~0-001-amber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=quick-turn-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~22-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8264408215759&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=feeler%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8264408215759%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001457770%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:34:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/feeler-gauges/feeler-gauge-thicknesses-included~0-001-amber/ 2025-11-02 08:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=r321%252f38-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=crowfeet/size~1-9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2083A46&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222083A46%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8893K195&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228893K195%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30138 pages (at 52 pages/min), scraped 894 items (at 1 items/min) 2025-11-02 08:35:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~80-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/crowfeet/size~1-9-16/ 2025-11-02 08:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~22-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cups/material~steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=current-meters/display-type~digital&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gauge-blocks/size~22-mm/ 2025-11-02 08:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=r399-35-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~12-6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9963305174922&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cups&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9963305174922%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8893K195&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228893K195%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~80-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5141729687232&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=current%20meters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5141729687232%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001939551%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=r8-collets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-control-devices&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/gauge-blocks/size~80-mm/ 2025-11-02 08:35:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~12-6mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feeler-gauges/thicknesses-included~5-0-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8264408215759&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=feeler%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8264408215759%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001455539%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-cylinders/pipe-size~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/diameter~12-6mm/ 2025-11-02 08:35:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=personnel-rescue-tripods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=custom-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~18-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2083A59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222083A59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutoff-wheels/abrasive-material~diamond&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~285/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5171308755878&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hydraulic%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5171308755878%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000067115%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30187 pages (at 49 pages/min), scraped 895 items (at 1 items/min) 2025-11-02 08:36:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feeler-gauges/thickness~5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=cutoff-wheels/diameter~1-1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522125%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=765203497970&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cutoff%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B765203497970%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548846%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1304K21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221304K21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/feeler-gauges/thickness~5mm/ 2025-11-02 08:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/feet/mount-type~hammer-on/mount-type~tack/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=765203497970&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cutoff%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B765203497970%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000053987%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2083A59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222083A59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~2-25-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~285&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=765203497970&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cutoff%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B765203497970%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548846%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~285/ 2025-11-02 08:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=racks/component~gear-rack&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauss-testers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feet/mount-type~hammer-in&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2083A59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222083A59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~2-25-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~3175g/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=racks/component~gear&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=racks/component~internal-gear&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=684483403398&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=racks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B684483403398%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002237364%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/feet/mount-type~hammer-in/ 2025-11-02 08:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536268%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=765203497970&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=cutoff%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B765203497970%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548846%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=684483403398&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=racks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B684483403398%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002260682%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=684483403398&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=racks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B684483403398%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002264006%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6832K73&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226832K73%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-tees/material~plastic-1/material~pfa-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauss-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:36:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:37:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~3175g&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7880K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227880K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2696N5&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222696N5%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30238 pages (at 51 pages/min), scraped 897 items (at 2 items/min) 2025-11-02 08:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536268%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~3175g/ 2025-11-02 08:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:12 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~327/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pfa-pipe-tees&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gear-box-vents/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-reservoirs/power-source~gravity&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~3457nall/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~3-75-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536268%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7305746640836&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hydraulic%20reservoirs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7305746640836%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076310%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feet/thread-size~5-8-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gear-box-vents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=feet/thread-size~m18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~3457nall&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8031429930803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=feet&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8031429930803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453021%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~3-75mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:46 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8031429930803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=feet&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8031429930803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334699%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~3457nall/ 2025-11-02 08:37:51 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=dampers/material~plastic-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/diameter~3-75mm/ 2025-11-02 08:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/willow-glass/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glare-reducing-aerosol-coatings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3077N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223077N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=idler-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/phillips-screwdriver-bit-sockets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9994033470274&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=dampers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9994033470274%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30287 pages (at 49 pages/min), scraped 899 items (at 2 items/min) 2025-11-02 08:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:11 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=development-boards/memory-size~1-kb&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass/super-flexible-corning-willow-glass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glare-reducing-aerosol-coatings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fence-panel-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indexers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glove-bags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=phillips-screwdriver-bit-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979761309988&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=development%20boards&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979761309988%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002216101%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~74/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9732413836877&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9732413836877%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%229763068013261%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=din-rail-mounting-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1387N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221387N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6705N111&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226705N111%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~9-7-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glove-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~74&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rail-guides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rail-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indicators/for-use-with~screws-and-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1387N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221387N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~74/ 2025-11-02 08:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/goggle-headbands/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~9-7mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/fabrication~fired/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=displays/display-type~scale&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979599325048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979599325048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965671%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/picks/tip-style~hook/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/diameter~9-7mm/ 2025-11-02 08:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/material~leather-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30336 pages (at 49 pages/min), scraped 901 items (at 2 items/min) 2025-11-02 08:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9962059121163&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=displays&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9962059121163%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001933428%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1387N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221387N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=goggle-headbands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~9005-100w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=doors/material~copper-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/fabrication~fired&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=picks/tip-style~hook&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/material~leather&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=double-coated-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1093630895265&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=doors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1093630895265%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000083208%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002259992%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:30 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pig-mats/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5551314863993&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5551314863993%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002275736%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/material~leather/ 2025-11-02 08:39:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~9005-100w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:36 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=doors/material~copper-1&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:39:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rails/material~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~923/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~9005-100w/ 2025-11-02 08:39:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/thread-size~1-7-8-8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pig-mats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fep-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferraz-shawmut-type-atm-r-fuses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrite-cores&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rails/material~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~923&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~1-7-8-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~btr/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rails/material~fiberglass/ 2025-11-02 08:40:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~923/ 2025-11-02 08:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30382 pages (at 46 pages/min), scraped 901 items (at 0 items/min) 2025-11-02 08:40:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rods/thread-size~1-7-8-8/ 2025-11-02 08:40:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~e211-2r/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/thread-size~m14/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~btr&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~9007/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=raised-face-slip-on-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inserts/thread-size~m20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rakes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/fitting-type~nut&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~e211-2r&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~btr/ 2025-11-02 08:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9893535896715&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=inserts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9893535896715%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~m14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002465810%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~e211-2r/ 2025-11-02 08:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~exy/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grinding-workstations/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334532%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~9007&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pin-in-torx-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~btl/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~9007/ 2025-11-02 08:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~f48t12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gripple-rope-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/for-use-with~formic-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-workstations&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/for-use-with~iron-chloride&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~exy&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:40:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pincers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964887%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968005%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~exy/ 2025-11-02 08:41:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~btl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30429 pages (at 47 pages/min), scraped 901 items (at 0 items/min) 2025-11-02 08:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~f48t12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gripple-rope-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/length~0-718&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-in-torx-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7436T711&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227436T711%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7436T711&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227436T711%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~btl/ 2025-11-02 08:41:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~f48t12/ 2025-11-02 08:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pincers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001584985%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~fp24-ho/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ratchet-wrenches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~h1-130w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/roller-chain-master-links/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5349K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225349K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/performance~moisture-resistant-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~fp24-ho&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insulation&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002086534%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/length~2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~h1-130w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roller-chain-master-links&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ferrules/length~25-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5349K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225349K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~fp24-ho/ 2025-11-02 08:41:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:41:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001593106%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:41:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~h1-130w/ 2025-11-02 08:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=internal-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001604169%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/laundry-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~pr30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hand-held-electrical-enclosures/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30480 pages (at 51 pages/min), scraped 903 items (at 2 items/min) 2025-11-02 08:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5081K95&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225081K95%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5349K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225349K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=inventory-bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001593106%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=razor-blades/for-cutting~sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~pr7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9068524016690&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=razor%20blades&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9068524016690%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930493%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~pr30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-held-electrical-enclosures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:29 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hand-held-metal-detectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/bulb-trade-number~q50t4-cl/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6518880407998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ferrules&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6518880407998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001593106%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~pr30/ 2025-11-02 08:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~pr7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=irrigation-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~pr7/ 2025-11-02 08:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hand-held-tachometers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-held-metal-detectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/trade-number~q50t4-cl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:42:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamer-guides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hand-presses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/iso-roller-chain-and-links/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:42:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/trade-number~q50t4-cl/ 2025-11-02 08:42:58 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamer-sets/material~cobalt-steel/reamer-type~for-dowel-pin-holes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-held-tachometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/voltage~130v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-diameter~35-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30526 pages (at 46 pages/min), scraped 904 items (at 1 items/min) 2025-11-02 08:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jack-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-diameter~42-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7218065360791&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamer%20sets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7218065360791%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512485%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003458002%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-presses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003552894%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:19 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamer-sets/material~cobalt-steel/reamer-type~for-dowel-pin-holes&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:43:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/iso-roller-chain-and-links/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jacks/power-source~manual&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K226&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K226%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-diameter~35mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hand-wipes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-diameter~42mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9965639372038&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=jacks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9965639372038%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076424%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamers/taper-pin-number~1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rollers/roller-diameter~35mm/ 2025-11-02 08:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rollers/roller-diameter~42mm/ 2025-11-02 08:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/voltage~2-7v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/voltage~24v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hand-wipes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 08:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 08:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hanging-document-files/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003552958%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/for-taper-pin-number~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/voltage~6-3v-dc/voltage~6-3v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jewelers-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K563&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K563%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reamers/for-taper-pin-number~1/ 2025-11-02 08:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553680%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553638%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fh-632-4-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:43:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/voltage~2-7v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~14-9-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hanging-document-files&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/material~steel-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reamer-sets/system-of-measurement~metric/material~high-speed-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30570 pages (at 44 pages/min), scraped 906 items (at 2 items/min) 2025-11-02 08:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/drill-bit-size~9-ga-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-1255/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~0-25-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~14-9-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:21 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 08:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:21 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 08:44:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-141/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/metric-size-high-speed-steel-reamer-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~0-38-w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000170038%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-1255&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hastelloy-x/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/drill-bit-size~9-ga-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~0-25w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reamers/metric-size-high-speed-steel-reamer-sets/ 2025-11-02 08:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001491991%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001535357%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/drill-bit-size~9-ga-/ 2025-11-02 08:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2276T64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222276T64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-141&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~0-25w/ 2025-11-02 08:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7890K1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227890K1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hastelloy-x&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A465&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A465%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529959%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hazardous-location-light-fixtures/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=junction-box-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~0-74-w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:01 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001491187%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30619 pages (at 49 pages/min), scraped 909 items (at 3 items/min) 2025-11-02 08:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~1-4-w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~12-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-198/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529959%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hazardous-location-light-fixtures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8034K56&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228034K56%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001491700%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-205/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keyed-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9527T022&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229527T022%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~12-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-198&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-207/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001529959%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~26-3-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-205&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001628965%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000106270%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~0-212/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keyless-hubs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~27-5-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001524608%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:49 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-198&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-207&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~26-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001628965%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:45:58 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-205&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~29-19-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-212&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001545062%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000068918%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~27-5-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30666 pages (at 47 pages/min), scraped 911 items (at 2 items/min) 2025-11-02 08:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548780%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0720&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:11 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-207&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~12-8-w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7535N141&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227535N141%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000005992%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001628965%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~29-19-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:19 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-212&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0795&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000088421%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001491196%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2312T66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222312T66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000029078%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000022214%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:32 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0720&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K374&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K374%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7535N141&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227535N141%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:36 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-(frp)-angles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2312T66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222312T66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~2-25-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000155569%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0845&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7535N141&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227535N141%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:46:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-(frp)-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~35-19-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2478K105&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222478K105%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000071243%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-bar-grating&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2312T66&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222312T66%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30717 pages (at 51 pages/min), scraped 913 items (at 2 items/min) 2025-11-02 08:47:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~2-25-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:11 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:47:11 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-0845&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:47:13 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~20-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~35-19-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~1-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-width~8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001627100%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:24 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~5-0000-mm/diameter~5-000-mm/diameter~5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/length~20-1/ 2025-11-02 08:47:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000145964%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~3-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rolling-ladders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/end-type~wire-retainer/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~8mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001629272%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~27-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rollers/roller-width~8mm/ 2025-11-02 08:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fiberglass-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001612166%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rolling-ladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/end-type~wire-retainer&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:47:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pins/diameter~5mm/ 2025-11-02 08:47:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001547051%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:00 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~30&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001927954%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~27w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cotter-pins/cotter-pin-type~hammer-set/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30764 pages (at 47 pages/min), scraped 914 items (at 1 items/min) 2025-11-02 08:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:12 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/end-type~wire-retainer&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~27w/ 2025-11-02 08:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2277T59&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222277T59%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laboratory-stoppers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~5-5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fibrous-rope-compression-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ladder-wall-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lag-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lags/fastener-head-type~hex/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/hammer-set-cotter-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001616358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pins/hammer-set-cotter-pins/ 2025-11-02 08:48:33 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:48:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:48:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/hex-drivers/size~no-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~no-19/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~8-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/overall-length~82-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lags/head-type~hex&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~t50/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-head-thumb-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~7-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~36-1-w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/lags/head-type~hex/ 2025-11-02 08:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~no-19&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001597727%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1024807243217&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20bulbs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1024807243217%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001491348%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~8-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/hex-drivers/size~no-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~3-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/length~82mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~t50&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-head-thumb-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~no-19/ 2025-11-02 08:49:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1505K383&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221505K383%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001603996%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/length~82mm/ 2025-11-02 08:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30812 pages (at 48 pages/min), scraped 914 items (at 0 items/min) 2025-11-02 08:49:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/hex-drivers/size~t50/ 2025-11-02 08:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~4-2-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/pin-type~dowel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~3w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/drill-bits/material~ceramic-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:20 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 08:49:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 08:49:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=file-storage-cabinets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~3w/ 2025-11-02 08:49:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=latch-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rope/diameter~1-4-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=files/abrasive-material~novaculite&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~4-2w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/number-of-pieces~15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/pin-type~dowel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/material~ceramic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7794418569789&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=files&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7794418569789%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001530140%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~4-2w/ 2025-11-02 08:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-pressure-hose-reels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001538867%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969599%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/drill-bits/material~ceramic/ 2025-11-02 08:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1885&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rope/diameter~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:48 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=files/abrasive-material~novaculite&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rope/diameter~7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:51 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/number-of-pieces~15&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:49:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000106443%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/files/for-use-on~rubber-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-1965&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:49:56 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rope/diameter~1-4/ 2025-11-02 08:49:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulbs/wattage~7-w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-temperature-felt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-pressure-hose-reels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000096680%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rope/diameter~7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30859 pages (at 47 pages/min), scraped 915 items (at 1 items/min) 2025-11-02 08:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rope/material~polyester-elastomer&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000106443%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=files/for-use-on~rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7090012887875&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7090012887875%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000149734%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~11-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/rope/material~polyester-elastomer/ 2025-11-02 08:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulbs/wattage~7w&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-temperature-felt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/files/for-use-on~rubber/ 2025-11-02 08:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/thread-size~10-24/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000051781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/light-bulbs/wattage~7w/ 2025-11-02 08:50:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rope-hooks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000106443%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lcr-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~14-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/thread-size~10-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~13&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rope-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fillers/for-joining~abs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/high-voltage-tape/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000083017%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000097288%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fillers/for-joining~cast-iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5826054618903&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fillers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5826054618903%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002452174%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5826054618903&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fillers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5826054618903%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454217%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30910 pages (at 51 pages/min), scraped 915 items (at 0 items/min) 2025-11-02 08:51:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=high-voltage-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-meters/light-technology~mercury-vapor-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rotary-locks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=leashes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-starters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~182-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9175013334974&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=light%20meters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9175013334974%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001968811%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2555&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:51:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000093312%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:31 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-meters/light-technology~mercury-vapor-1&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hochdruck-gaskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334743%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=letter-jobbers-drills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rotating-joint-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000156820%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hollow-lock-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:47 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2555&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~335-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hochdruck-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:53 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/homelite-compatible-chain-saw-chains/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-end-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotating-joint-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hollow-lock-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000110871%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=film-sealers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 30960 pages (at 50 pages/min), scraped 915 items (at 0 items/min) 2025-11-02 08:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filter-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=homelite-compatible-chain-saw-chains&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-end-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2905&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/round-head-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2935&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levers/handle-type~fixed&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2980&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000170091%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/for-use-with~ammonia/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000130453%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/for-use-with~fuel-oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978677157349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=levers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978677157349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001932515%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000138279%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:36 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2905&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-head-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filter-sleeves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:40 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2935&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:52:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:42 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=levers/handle-type~fixed&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:52:43 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-2980&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/round-plugs-for-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~anhydrous-ammonia&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~fuel-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pipe-fittings/for-use-with~anhydrous-ammonia/ 2025-11-02 08:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=drill-bits/overall-length~86-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3200&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3270&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-plugs-for-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lint-traps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2657592637062&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=drill%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2657592637062%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000096805%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31009 pages (at 49 pages/min), scraped 915 items (at 0 items/min) 2025-11-02 08:53:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055909%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000097658%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3281&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/round-head-drilling-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:17 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3200&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:53:18 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3270&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lip-and-spur-drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-troughs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filters/for-reducing~sodium-chloride/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000094747%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/for-use-with~sodium-carbonate-soda-ash/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/duct/for-use-with~liquids/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:53:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/for-use-with~grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hour-meters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-head-drilling-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-troughs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000094747%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/for-use-with~sodium-carbonate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968046%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=liquids-duct&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filters/for-reducing~sodium-chloride/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pipe-fittings/for-use-with~sodium-carbonate/ 2025-11-02 08:53:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98355K833&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298355K833%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/huck-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hour-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rtv-silicone-adhesives/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000094747%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/humidity-control-strip-heaters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31055 pages (at 46 pages/min), scraped 916 items (at 1 items/min) 2025-11-02 08:54:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 08:54:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3585&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 08:54:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=huck-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3630&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rtv-silicone-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=humidity-control-strip-heaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086513%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=limit-switches/environment~outdoor&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000067062%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hvac-thermostats/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=limit-switches/environment~corrosive&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:27 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3585&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/hvac-filter-trade-size~10-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2001310163167&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=limit%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2001310163167%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086967%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:34 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3630&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2001310163167&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=limit%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2001310163167%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000055772%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002068002%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=line-pulleys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3760&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hvac-thermostats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/hvac-filter-trade-size~14-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7988K6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227988K6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=linear-ball-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000077574%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/hvac-filter-trade-size~15-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002056014%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:54:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3810-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:01 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3760&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:55:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:55:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002065926%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hydraulic-jack-set-manifolds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000147677%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31108 pages (at 53 pages/min), scraped 917 items (at 1 items/min) 2025-11-02 08:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/hvac-filter-trade-size~18-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=linear-motion-actuators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=load-rated-hold-open-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:15 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-3810-1&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002066955%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/hvac-filter-trade-size~25-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lobed-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locating-pin-retractors/power-source~air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filters/material~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-jack-set-manifolds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002056077%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/hvac-filter-trade-size~6-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filters/material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-cords/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3355991442690&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=locating%20pin%20retractors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3355991442690%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076415%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/fitting-type~union-1/finish~galvanized/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/i-beam-tube-clamps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9862682945404&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9862682945404%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002057665%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/material~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/coffee-filters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/material~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3355991442690&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=locating%20pin%20retractors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3355991442690%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076415%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/filters/material~iron/ 2025-11-02 08:55:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/type~union/finish~galvanized&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4200&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=i-beam-tube-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:55:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/filters/material~plastic/ 2025-11-02 08:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-feet/mount-type~press-fit/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550956%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/paper-coffee-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000117303%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31157 pages (at 49 pages/min), scraped 917 items (at 0 items/min) 2025-11-02 08:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-food-grade-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/fitting-type~union-1/pipe-size~1-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3355991442690&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=locating%20pin%20retractors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3355991442690%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076415%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4315&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/filters/paper-coffee-filters/ 2025-11-02 08:56:17 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4200&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-feet/mount-type~press-fit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filters/hvac-filter-trade-size~18-18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000046429%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-food-grade-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-fittings/type~union/pipe-size~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:56:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8031429930803&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8031429930803%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:33 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4315&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4415&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4425&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-flange-gaskets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7983833808297&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7983833808297%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000067115%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/trade-size~18-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locknuts/military-specification~ms21044-n04/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000162161%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000043808%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/impulse-sealers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-toggle-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/filters/trade-size~18-18/ 2025-11-02 08:56:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-floor-flanges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:55 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4425&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-flange-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locknuts/military-specification~ms21045-9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/index-card-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locknuts/military-specifications-met~ms21044-n04&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000162161%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/filters/hvac-filter-trade-size~48-72/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=impulse-sealers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31202 pages (at 45 pages/min), scraped 917 items (at 0 items/min) 2025-11-02 08:57:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-floor-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/locknuts/military-specifications-met~ms21044-n04/ 2025-11-02 08:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/indicating-silica-gel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4575&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locknuts/military-specifications-met~ms21045-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=index-card-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000162161%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber-round-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=filters/trade-size~48-72&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000113337%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/locknuts/military-specifications-met~ms21045-9/ 2025-11-02 08:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/industrial-telephones/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/filters/trade-size~48-72/ 2025-11-02 08:57:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4685&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indicating-silica-gel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:37 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4575&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 08:57:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-round-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000012074%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locks/for-use-on~glass-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=industrial-telephones&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locks/lock-type~slide-to-release&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-labels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:57:56 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4685&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:57:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971224262213&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=locks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971224262213%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929681%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4820&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971224262213&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=locks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971224262213%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001974594%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locksmiths-files/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=finger-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=11125A13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2211125A13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubbermaid-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31247 pages (at 45 pages/min), scraped 917 items (at 0 items/min) 2025-11-02 08:58:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086756%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:17 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4820&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-97483&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/instrument-knobs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locksmiths%27-files&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4905&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-tap-and-die-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubbermaid-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-5550&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000104854%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation-supports/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=instrument-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000056632%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation-testers-with-nist-certificate/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-tap-and-die-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=logging-chain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:58:50 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-5550&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:58:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:54 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-4905&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loto-(lockout%252ftagout)-lockouts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation-supports&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-vises/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation-testers-with-nist-certificate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/international-plug-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loto-(lockout-tagout)-lockouts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-wrap-insulation/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31295 pages (at 48 pages/min), scraped 918 items (at 1 items/min) 2025-11-02 08:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-6860&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loop-to-loop-lanyards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-vises&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000006286%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rulers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=international-plug-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pipe-wrap-insulation&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=low-profile-flange-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:27 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 08:59:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=low-pressure-water-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:28 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~0-6860&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 08:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-resistant-coveralls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/invoice-envelopes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rulon-sheets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~1-1250&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pivot-hinges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~1-3125&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=invoice-envelopes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-retardant-silicone-foam&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/for-use-on~precision-measurement-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fire-seal-weatherstripping&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulon-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 08:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038441%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000156754%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929720%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31345 pages (at 50 pages/min), scraped 918 items (at 0 items/min) 2025-11-02 09:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pivot-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:12 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/reamer-diameter~1-3125&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 09:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/firefighters-axes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plasma-cutting-supplies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2278N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222278N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/oil-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plaster-of-paris/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038441%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plain-bearings/ultra-low-friction-dry-running-sleeve-bearings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%225479482071661%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jet-pallet-truck-replacement-parts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=firefighters%27-axes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lumber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plain-bearings/ultra-low-friction-dry-running-sleeve-bearings/ 2025-11-02 09:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plasma-cutting-supplies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2278N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222278N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2158K34&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222158K34%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plaster-of-paris&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038441%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jet-pallet-truck-replacement-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jewelers-loupes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=m12-cordsets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2278N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222278N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/jewelers-slotting-saw-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:00:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:00:58 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=first-aid-guides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=m4-buttons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31393 pages (at 48 pages/min), scraped 919 items (at 1 items/min) 2025-11-02 09:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/saddle-washers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reamers/shank-type~l-handle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~lr-v08/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jewelers%27-loupes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:13 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:01:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jewelers%27-slotting-saw-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002128079%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fish-tape-leaders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saddle-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-cleaners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lr-v08-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricant-analysis-kits/for-testing~hydraulic-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricant-analysis-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002128079%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-screw-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-coated-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9976959188459&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricant%20analysis%20kits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9976959188459%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002067314%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sae-thread-ball-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2176N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222176N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-containers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=receiving-desks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5781696663338&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reamers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5781696663338%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002128079%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/receptacles/nema-type~l10-20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-tool-taps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-coated-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31440 pages (at 47 pages/min), scraped 920 items (at 1 items/min) 2025-11-02 09:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/safety-barriers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sae-thread-ball-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-containers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/key-panel-actuators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=receptacles/nema-style~l10-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/for-use-on~ceramic-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fixture-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-drawstring-bags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-barriers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machinery-spacers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/receptacles/nema-style~l10-20/ 2025-11-02 09:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/safety-fencing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-dust-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/formulation~neatsfoot-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/kick-plates/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flags/u-s-state-territory~hawaii/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/safety-glasses-lenses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-drawstring-bags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092946%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flags/u-s-state-territory~washington/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-fencing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnifiers/magnification~3-5x&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-dust-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1022K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221022K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=kick-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flags/state-us-territory~hawaii&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-glasses-lenses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31487 pages (at 47 pages/min), scraped 920 items (at 0 items/min) 2025-11-02 09:03:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9673273568857&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=magnifiers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9673273568857%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081206%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flags/state-us-territory~washington&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/flags/state-us-territory~hawaii/ 2025-11-02 09:03:16 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/thickener~polyurea&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6611N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226611N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/flags/state-us-territory~washington/ 2025-11-02 09:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/safety-snap-pins/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1022K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221022K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~dri-slide-multi-purpose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/laboratory-lifts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002073797%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-luer-lock-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manifold-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002085440%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2303N1&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222303N1%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-snap-pins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1022K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221022K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laboratory-lifts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flame-resistant-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-luer-lock-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flammable-liquids-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/fitting-connection~push-to-connect-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:00 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/fitting-type~connector-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-relief-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=material-racks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31538 pages (at 51 pages/min), scraped 922 items (at 2 items/min) 2025-11-02 09:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002362219%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flange-mounted-bushings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/for-use-with~glass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~never-seez-regular-grade&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001919498%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~slick-50-onelube&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136502%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~hydrochloric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-relief-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~glass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002100998%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fomblin-lubricants/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002963068%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-sheets/color~gold/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/for-use-with~natural-gas-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~wd-40-big-blast&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reducers/for-use-with~glass/ 2025-11-02 09:04:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=11785K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2211785K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967579%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/laser-beam-mirrors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108421%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/vacuum-seal-and-o-ring-grease/thickener~ptfe&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/material~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sandpaper-flap-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-sheets/color~gold&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=medium-screw-in-base-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002116843%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5968784707760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5968784707760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001549824%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=laser-beam-mirrors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108421%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=10195K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2210195K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/material~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31589 pages (at 51 pages/min), scraped 923 items (at 1 items/min) 2025-11-02 09:05:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mechanical-jack-lifts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sandpaper-flap-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/material~titanium-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reducers/material~fiberglass/ 2025-11-02 09:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lathe-tool-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108421%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/reducers/thread-size~1-3-4-8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-sinks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=10195K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2210195K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/flange-od~20-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/material~titanium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lathe-tool-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000082664%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machinable-shaft-couplers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:40 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:05:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reducers/material~titanium/ 2025-11-02 09:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-cages&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/thread-size~1-3-4-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lavatory-sinks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=10195K65&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2210195K65%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6806K414&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226806K414%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sanitary-tube-fittings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/reducers/thread-size~1-3-4-8/ 2025-11-02 09:05:57 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lead-holders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducers/thread-size~8-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:08 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31638 pages (at 49 pages/min), scraped 924 items (at 1 items/min) 2025-11-02 09:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/left-hand-threaded-thin-hex-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/for-use-with~fuel-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-slip-on-structural-framing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000060837%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-waste-containers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-button-head-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lead-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-u-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=left-hand-threaded-thin-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-slip-on-structural-framing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000060837%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/legend-plates/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/socket-head-cap-screws/system-of-measurement~metric/fastener-strength-grade-class~iso-class-10-9/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lens-cleaning-towelettes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lepages-glue/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducing-crosses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reducing-pipe-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9981614637306&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=reducers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9981614637306%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000060837%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=legend-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-class-109-steel-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lens-cleaning-towelettes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982323229449&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flanges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982323229449%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-flanged-hex-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flanges/specifications-met~asme-standard/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lepage%27s-glue&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31686 pages (at 48 pages/min), scraped 924 items (at 0 items/min) 2025-11-02 09:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:09 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/saws/for-use-on~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastic-tags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lift-rite-pallet-truck-replacement-parts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-hex-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:07:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lift-rite-truck-wheels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machinists-jacks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flanges/specifications-met~asme&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tape-rulers/system-of-measurement~metric/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saws/for-use-on~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastic-tags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/flanges/specifications-met~asme/ 2025-11-02 09:07:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lift-rite-pallet-truck-replacement-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/saws/for-use-on~aluminum/ 2025-11-02 09:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/saws/for-use-on~tool-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lift-rite-truck-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machinists%27-jacks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machinists-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-tape-rulers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/light-bulb-receptacles/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=refrigerant-recovery-bottles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saws/for-use-on~tool-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saws/miter-chop-and-table-saw-blades-for-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-air-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=661583884380&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B661583884380%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930358%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=light-bulb-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=661583884380&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B661583884380%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%228785306125131%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31735 pages (at 49 pages/min), scraped 924 items (at 0 items/min) 2025-11-02 09:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-bit-drivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=refrigerator-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flash-drives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4734N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224734N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mil-spec-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6909A46&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226909A46%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=milling-heads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cleaners/for-removing~scale/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/regulators/for-use-with~hydraulic-fluid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plastics/thickness~1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-cup-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miniature-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/liquid-shipping-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-key-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-nutdriver-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scale-cleaners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-pens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=regulators/for-use-with~hydraulic-fluid-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plastics/thickness~1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/littelfuse-type-225-fuses/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=liquid-shipping-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/regulators/for-use-with~hydraulic-fluid-1/ 2025-11-02 09:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/scrapers/blade-shape~curved/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mixer-repair-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=778212328374&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B778212328374%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001455539%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/regulators/inlet-pipe-size~1-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31783 pages (at 48 pages/min), scraped 926 items (at 2 items/min) 2025-11-02 09:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetically-actuated-safety-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flat-top-conveyor-chain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=littelfuse-type-225-fuses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scrapers/blade-shape~curved&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screen-discs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=regulators/inlet-pipe-size~3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=regulators/inlet-pipe-size~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3182438425377&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3182438425377%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000044019%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flat-wire&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plug-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7367171817318&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=regulators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7367171817318%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000080294%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/regulators/inlet-pipe-size~1/ 2025-11-02 09:09:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnets/magnet-grade~c5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flatness-gauges/manufacturer-model-number~81320&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screen-discs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reinforced-retaining-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locking-ball-joints/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5600408034591&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flatness%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5600408034591%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003447918%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3182438425377&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3182438425377%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000044019%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plug-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:09:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnets/grade~c5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/relays/switch-designation~4pdt/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=relay-burnishers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/pipe-size~1-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-ball-joints&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/magnets/grade~c5/ 2025-11-02 09:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=relay-timers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31831 pages (at 48 pages/min), scraped 926 items (at 0 items/min) 2025-11-02 09:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3182438425377&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3182438425377%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000044019%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/loctite-metal-repair-epoxies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flexible-duct-hose-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=relays/industry-designation~4pdt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mountable-cable-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/for-pipe-size~1-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/low-pressure-stainless-steel-unthreaded-flanges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/relays/industry-designation~4pdt/ 2025-11-02 09:10:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:10:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mounting-arms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plugs/for-pipe-size~1-16/ 2025-11-02 09:10:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:10:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-metal-repair-epoxies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=moving-slides&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/plug-connection-method~push-in/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plugs/thread-type~nptf/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/carton-staplers/power-source~manual/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/plumbers-grease/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=low-pressure-stainless-steel-unthreaded-flanges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flexible-tool-pans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubrication-hand-pumps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/plug-type~push-in&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plugs/thread-type~nptf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=manual-carton-staplers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=plumbers%27-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=multipole-connector-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=removable-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/plugs/plug-type~push-in/ 2025-11-02 09:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1040952587880&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1040952587880%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000119580%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31880 pages (at 49 pages/min), scraped 926 items (at 0 items/min) 2025-11-02 09:11:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubrication-hand-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floats/for-use-with~caustic-fluid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pipe-fittings/fitting-type~repair-clamp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=marine-battery-cable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lumber-crayons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nas-1/specifications-met~nas620c416/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2394922483006&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=floats&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2394922483006%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179277%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=marine-epoxies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=repinnable-padlocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8154K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228154K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=repair-clamp-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=markers/color~blue-white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-620-c416-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/marines-flags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=replacement-parts-for-cm-valustar-electric-chain-hoists&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9853367128756&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=markers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9853367128756%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002191248%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/floodlights/environment~hazardous/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screw-jacks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/made-to-order-extension-springs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=markers/color~orange-white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pneumatic-grease-guns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nickel/material~800-nickel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=machine-screw-jacks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31924 pages (at 44 pages/min), scraped 927 items (at 1 items/min) 2025-11-02 09:12:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floodlights/environment-1~hazardous&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=made-to-order-extension-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=resistivity-meters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9853367128756&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=markers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9853367128756%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002193306%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6971737856557&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nickel&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6971737856557%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005739734%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-finishes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/marines-flags/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/floodlights/environment-1~hazardous/ 2025-11-02 09:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-base-drill-hole-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:22 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-grease-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/connects-to~tube-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pneumatic-scrapers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/respirator-masks/niosh-approval-number~tc-84a-2006/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-dial-indicators/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-base-drill-hole-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirator-filter-retainers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~drinking-water&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:41 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:12:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/connects-to~tube&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964650%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pneumatic-scrapers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-mats/mat-type~entrance&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirator-masks/niosh-approval~tc-84a-2006&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-dial-indicators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nipples/connects-to~tube/ 2025-11-02 09:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/multipurpose-hose/material~buna-n/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/finish~cadmium-plated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:13:01 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3911241974115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=floor%20mats&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3911241974115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001974226%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-mats/mat-type~sticky-surface&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/respirator-masks/niosh-approval~tc-84a-2006/ 2025-11-02 09:13:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 31968 pages (at 44 pages/min), scraped 927 items (at 0 items/min) 2025-11-02 09:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/finish~gold-plated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=62315T69&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2262315T69%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3911241974115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=floor%20mats&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3911241974115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001973444%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nitrile-multipurpose-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/finish~cadmium-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=masks/respirator-filter-rating~n100&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-ramps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6896T61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226896T61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=floor-stickers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539197%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonwoven-abrasives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/finish~gold-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-v-blocks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3028195520048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=masks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3028195520048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000164327%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnetic-viewing-film/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001549195%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5450T28&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225450T28%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flags/u-s-state-territory~florida/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pole-lights/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hammer-retainers/compatible-tools~any-florida-pneumatic-model/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6896T61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226896T61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-v-blocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:13:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=material-support-stands/mobility~portable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001549195%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnetic-viewing-film&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5450T28&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225450T28%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=florida-flags&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=florida-pneumatic-compatible-air-hammer-retainers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pole-lights&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6896T61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226896T61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32015 pages (at 47 pages/min), scraped 928 items (at 1 items/min) 2025-11-02 09:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6441994498981&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=material%20support%20stands&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6441994498981%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000161231%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mail-organizers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001549195%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/clothing/application~flotation/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/poles/material~kevlar-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mailbox-pedestals/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5450T28&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225450T28%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:26 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:14:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mail-organizers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mats/for-use-with~carpet&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/manufacturer-model-number~6001&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:39 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:14:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:14:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flotation-clothing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=poles/material~kevlar&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mailbox-pedestals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1629427390652&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mats&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1629427390652%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984368%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/manufacturer-model-number~60921&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002126980%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/makita-compatible-staples/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/flow-controls/fitting-connection~luer-lock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:52 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/poles/material~kevlar/ 2025-11-02 09:14:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2848T12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222848T12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002114896%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5834T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225834T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polishing-sticks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=maximum-occupancy-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5834T22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225834T22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32063 pages (at 48 pages/min), scraped 930 items (at 2 items/min) 2025-11-02 09:15:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=makita-compatible-staples&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-controls/connection-style~quick-turn&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/niosh-approval~tc-23c-3088&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nutdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polycarbonate-bar-stock/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polishing-sticks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/flow-controls/connection-style~quick-turn/ 2025-11-02 09:15:21 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:15:21 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/respirators/niosh-approval~tc-23c-3088/ 2025-11-02 09:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/nut-type~locknut/thread-size~1-4-28-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=respirators/niosh-approval-number~tc-21c-0836&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/nut-type~locknut/thread-type~unf-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polycarbonate-bar-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/material-handling-forks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-controls/for-use-with~liquid-argon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9152409968949&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=respirators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9152409968949%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002066583%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~3-4-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=menstrual-hygiene-disposal-bag-dispensers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~1-4-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9072876622555&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20controls&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9072876622555%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968165%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2428N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222428N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334890%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-type~unf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~0-855-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/thread-size~m20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=material-handling-forks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=49415K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2249415K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:15:57 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~3-4-10&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 09:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455475%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32110 pages (at 47 pages/min), scraped 932 items (at 2 items/min) 2025-11-02 09:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyethylene-anchors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=79215A413&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2279215A413%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~m20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retaining-rings/retaining-ring-style~side-mount&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=49415K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2249415K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~2-56&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-switches/body-material~noryl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6646950986674&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=retaining%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6646950986674%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002052814%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh/mesh-size~198-198&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retractable-air-reels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-anchors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000031867%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8779931053344&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8779931053344%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001573872%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97431A210&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297431A210%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3985461849490&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mesh&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3985461849490%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002065318%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=49415K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2249415K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retractable-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4646N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224646N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/retractable-spring-plungers/fastener-head-type~l-handle/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9318T47&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229318T47%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/formulation~buna-n/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylock-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:16:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retractors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mesh/mesh-size~86-86&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyethylene-rubber-strips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=retractable-spring-plungers/handle-type~l-handle&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/formulation~fluorine/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:17:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32161 pages (at 51 pages/min), scraped 936 items (at 4 items/min) 2025-11-02 09:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3985461849490&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mesh&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3985461849490%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002054148%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polyethylene-u-channels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-straps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3688794005080&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=retractable%20spring%20plungers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3688794005080%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001938773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reusable-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-rubber-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/formulation~fluorine&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3985461849490&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mesh&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3985461849490%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002054148%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polyethylene-u-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=reverse-pliers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-detectable-plastic-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145451%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~14300/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-detectable-uhmw-polyethylene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-valves/valve-type~ball/handle-type~lockable-lever&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flow-valves/valve-type~ball/pipe-size~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3985461849490&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mesh&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3985461849490%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002054148%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polypropylene-drum-plugs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=339191585183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B339191585183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247047%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001944518%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145451%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=339191585183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=flow%20valves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B339191585183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247047%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000050647%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~14300&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~14503&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-ratcheting-gears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4776K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224776K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3634K302&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223634K302%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:06 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32212 pages (at 51 pages/min), scraped 936 items (at 0 items/min) 2025-11-02 09:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145905%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sealants/manufacturer-model-number~14503/ 2025-11-02 09:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-drum-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/for-use-with~benzene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145451%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967302%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~295/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ribbed-neck-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145905%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ribbed-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/for-use-with~fluorinated-solvents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:32 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ridgid-770-die-head-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/for-use-with~phosphoric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179263%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polypropylene-string/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~295&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polypropylene-wire-ties/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145905%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002112214%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-clip-on-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/polystyrene-boxes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-brass-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-string&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6937T8&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226937T8%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/o-rings/for-use-with~phosphoric-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polypropylene-wire-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=right-angle-boots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32263 pages (at 51 pages/min), scraped 939 items (at 3 items/min) 2025-11-02 09:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=polystyrene-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-wrap/material~polyurethane-rubber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~594/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fluorescent-ballasts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-thumb-screw-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:23 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-steel-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mice&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-trays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pop-rivets/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~594&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flush-mount-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002132399%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slings/number-of-legs~1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flush-pems&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oilers/for-use-with~sulfuric-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/port-cutters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~795/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74945A67&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274945A67%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4227901971716&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oilers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4227901971716%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965374%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~8019/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~920/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=one-leg-slings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32308 pages (at 45 pages/min), scraped 940 items (at 1 items/min) 2025-11-02 09:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=port-cutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~795&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ring-gauges/thread-size~m12-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foam-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~8019&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002151134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=flyback-diodes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~dp605ns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~920&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002143555%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7452A21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227452A21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/porting-tools/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ring-gauges/thread-size~m12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002131134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74885A23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274885A23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:42 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ring-gauges/thread-type~uns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~dp605ns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:44 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/ring-gauges/thread-size~m12/ 2025-11-02 09:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7585A42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227585A42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlet-strips/iec-connector-type~c14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5131635916106&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=ring%20gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5131635916106%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107915%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~mr-2000/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002127379%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/sealants/manufacturer-model-number~el2000/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~pc-7222/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7687249081189&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlet%20strips&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7687249081189%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000114078%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-industry-handles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3160A463&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223160A463%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/post-it-notes/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/post-mounts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5511N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225511N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32354 pages (at 46 pages/min), scraped 943 items (at 3 items/min) 2025-11-02 09:21:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~mr-2000&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002127379%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~pc-7222&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/sealants/manufacturer-model-number~el2000/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/manufacturer-model-number~tl42/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3160A463&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223160A463%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002121034%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=post-it-notes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealants/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002100356%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7489A35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227489A35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=post-mounts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002127379%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7489A33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227489A33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/manufacturer-model-number~tl42&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3160A463&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223160A463%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:39 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:21:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overflow-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089207%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:56 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivet-nut-setter&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overhead-mixers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealers/for-use-on~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivet-nuts/finish~zinc-yellow-chromate-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/framing/material~rubber-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overload-clutches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32402 pages (at 48 pages/min), scraped 946 items (at 3 items/min) 2025-11-02 09:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7862604308299&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivet%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7862604308299%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001547790%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivet-nuts/thread-size~12-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mig-gun-nozzles/manufacturer-model-number~m-100&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/for-use-on~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7862604308299&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivet%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7862604308299%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334663%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-adapters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=framing/material~rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9740843502878&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mig%20gun%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9740843502878%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004784409%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mig-gun-nozzles/manufacturer-model-number~mdx-100&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sealers/for-use-on~aluminum/ 2025-11-02 09:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivet-spacers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7862604308299&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivet%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7862604308299%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001547790%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=98560A567&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2298560A567%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/framing/material~rubber/ 2025-11-02 09:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/binding-barrels-and-screws/material~stainless-steel-2/material~18-8-stainless-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9740843502878&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mig%20gun%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9740843502878%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004770200%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealers/for-use-on~plastic-1/for-use-on~abs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/foils/material~titanium-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealers/for-use-on~stone/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealers/manufacturer~devcon/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7862604308299&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivet%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7862604308299%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001547790%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealers/manufacturer~sika/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/18-8-stainless-steel-binding-barrels-and-screws-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9740843502878&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mig%20gun%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9740843502878%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004770200%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foils/material~titanium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealers/manufacturer~vibra-tite/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/for-use-on~stone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rivets/18-8-stainless-steel-binding-barrels-and-screws-9/ 2025-11-02 09:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/manufacturer~devcon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/manufacturer~sika&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/foils/material~titanium/ 2025-11-02 09:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32449 pages (at 47 pages/min), scraped 947 items (at 1 items/min) 2025-11-02 09:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-cords/voltage~24v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984350057510&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984350057510%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930162%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984350057510&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984350057510%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002263064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9740843502878&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mig%20gun%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9740843502878%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004770200%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984350057510&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984350057510%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002265862%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/manufacturer~vibra-tite&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74575A32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274575A32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mig-welding-liners/manufacturer-model-number~m-15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6937T35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226937T35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-cords/voltage~24v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9984350057510&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9984350057510%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002258464%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/diameter~0-151&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mig-welding-liners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1643606697087&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mig%20welding%20liners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1643606697087%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002325446%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:33 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4895046738192&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4895046738192%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003552958%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001546119%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealing-sheet-metal-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/diameter~0-175&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6937T35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226937T35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/power-cords/voltage~50v-ac/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=70235K54&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2270235K54%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97537A240&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297537A240%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001516643%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1643606697087&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mig%20welding%20liners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1643606697087%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002325446%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/diameter~0-191&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=garden-hose-fittings/thread-type~nh-nst&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealing-sheet-metal-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97358A108&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297358A108%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536272%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6937T35&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226937T35%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=power-cords/voltage~50v-ac&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32500 pages (at 51 pages/min), scraped 950 items (at 3 items/min) 2025-11-02 09:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4332774955340&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=garden%20hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4332774955340%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038314%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1643606697087&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mig%20welding%20liners&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1643606697087%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002325446%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97537A170&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297537A170%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4895046738192&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4895046738192%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003553748%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=garnet-sanding-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6494T22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226494T22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mil-g-3036-grommets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:24 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4302N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224302N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/color~beige&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:31 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/shaft-diameter~45-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97537A170&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297537A170%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/for-use-with~hydraulic-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/fastener-head-type~flat/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/for-use-with~abietic-acid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002203687%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964432%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-grade-pallets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/material~brass-1/fastener-head-type~domed&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-shaft-diameter~45mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=97537A170&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2297537A170%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/head-type~flat&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mil-spec-tube-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~abietic-acid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510572%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001585070%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:24:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/seals/for-shaft-diameter~45mm/ 2025-11-02 09:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rivets/head-type~flat/ 2025-11-02 09:25:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:25:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967239%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/calculators/environment~food-and-beverage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/shape~rectangular-tube&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32554 pages (at 54 pages/min), scraped 953 items (at 3 items/min) 2025-11-02 09:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/for-use-with~liquid-ammonia/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=90217A610&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2290217A610%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:09 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/for-use-with~natural-gas-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/for-use-with~propylene-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panel-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=77495A31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2277495A31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310680%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:24 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=food-industry-calculators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1520T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221520T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paper/for-use-on~copper-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~liquid-ammonia&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~natural-gas&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~propylene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/precision-control-knobs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paper/for-use-on~copper-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967188%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:39 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/seals/for-use-with~natural-gas/ 2025-11-02 09:25:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/seals/for-use-with~propylene/ 2025-11-02 09:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/precision-drive-shafts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/dial-type~continuous&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992232473973&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=paper&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992232473973%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930214%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/precision-extension-springs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paper/for-use-on~copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001930184%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-control-knobs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:25:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/paper/for-use-on~copper/ 2025-11-02 09:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/seals/material~zinc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miniature-drill-chucks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-drive-shafts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992232473973&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=paper&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992232473973%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930214%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32599 pages (at 45 pages/min), scraped 956 items (at 3 items/min) 2025-11-02 09:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/precision-shoulder-bolts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/precision-rulers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/thread-size~1-4-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/material~zinc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miniature-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/secure-hold-socket-extensions/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/thread-size~m10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512333%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9992232473973&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=paper&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9992232473973%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930214%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rivets/thread-size~m5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-shoulder-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=precision-rulers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000038594%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7107088629157&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rivets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7107088629157%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000011539%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=secure-hold-socket-extensions&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/system-of-measurement~letter-size/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/thread-type~unf-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miniature-screw-base-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:47 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/press-fit-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/security-sheet-metal-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sheets/material~borosilicate-glass/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gear-grease&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/system-of-measurement~letter&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/thread-type~unf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32647 pages (at 48 pages/min), scraped 956 items (at 0 items/min) 2025-11-02 09:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foot-operated-shears&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=press-fit-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauges/system-of-measurement~letter/ 2025-11-02 09:27:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:27:13 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=security-sheet-metal-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauges/thread-type~unf/ 2025-11-02 09:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass/borosilicate-glass-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass-and-ceramic-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rj-11-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/miter-saws/manufacturer~dewalt-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9732413836877&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glass&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9732413836877%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%224177099584392%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/self-clinching-studs/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8476K999&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228476K999%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robertson-screwdrivers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-alignment-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=miter-saws/manufacturer~dewalt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glides/mount-type~press-fit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/color~gray&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=foreline-traps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=self-clinching-studs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=photoelectric-detectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/miter-saws/manufacturer~dewalt/ 2025-11-02 09:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9952801634820&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glides&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9952801634820%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003010466%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/maximum-pressure~29-5-psi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gloves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001544617%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mixers/for-use-with~hydrogen-peroxide-90-concentration/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:02 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glides/mount-type~press-fit&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 09:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/self-tappers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/maximum-pressure~6000-psi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32694 pages (at 47 pages/min), scraped 957 items (at 1 items/min) 2025-11-02 09:28:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-cord-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-gauges/maximum-pressure~70-psi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/maximum-pressure-psi~29-5-psi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mixer-chucks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mixers/for-use-with~hydrogen-peroxide-90-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=forklift-jacks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=self-tappers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7534361604797&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7534361604797%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001740597%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/maximum-pressure-psi~6000-psi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/mixers/for-use-with~hydrogen-peroxide-90-/ 2025-11-02 09:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=forklift-mount-floor-brushes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-gauges/maximum-pressure-psi~70-psi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:28:33 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mixers/mixer-chucks-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/pressure-gauges/maximum-pressure-psi~6000-psi/ 2025-11-02 09:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7534361604797&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7534361604797%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001725530%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-safety-padding/for-robot-arm-manufacturer~staubli&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/mixers/mixer-chucks-6/ 2025-11-02 09:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pin-connectors/number-of-poles~8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-safety-padding/for-robot-arm-manufacturer~yaskawa-motoman&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9993803950476&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robot%20safety%20padding&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9993803950476%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247979%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobil-delvac-1230-diesel-engine-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9993803950476&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robot%20safety%20padding&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9993803950476%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002238349%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobil-delvac-1300-super-diesel-engine-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:28:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-connectors/number-of-poles~8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32745 pages (at 51 pages/min), scraped 957 items (at 0 items/min) 2025-11-02 09:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobil-dte-bb-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-safety-padding/for-robot-arm-manufacturer-series~mpp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9993803950476&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robot%20safety%20padding&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9993803950476%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247979%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobil-dte-10-excel-15-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pin-connectors/number-of-poles~8/ 2025-11-02 09:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-tanks/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-safety-padding/for-robot-arm-manufacturer-series~ob7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9993803950476&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robot%20safety%20padding&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9993803950476%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002152070%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobil-velocite-no-10-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-sensor-pads&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9993803950476&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robot%20safety%20padding&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9993803950476%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002134635%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9993803950476&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robot%20safety%20padding&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9993803950476%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002247979%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pressure-transmitters/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-teach-pendant-cord-reels/cord-manufacturer-equivalent-model-number~xgmf12573&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobile-device-cases/for-use-with~iphone-14-smartphones&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/price-tag-guns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2260525566009&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robot%20teach%20pendant%20cord%20reels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2260525566009%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002140964%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-b7-steel-bolts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:49 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/primary-cable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9937589136732&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mobile%20device%20cases&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9937589136732%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004152432%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pressure-transmitters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-teach-pendant-cords/for-robot-arm-manufacturer-model-number~10ia&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3821N104&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223821N104%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-teach-pendant-cords/manufacturer-equivalent-model-number~156798-3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/primers/color~white/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=price-tag-guns&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7442N33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227442N33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3733552981249&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robot%20teach%20pendant%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3733552981249%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002142075%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3733552981249&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robot%20teach%20pendant%20cords&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3733552981249%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005187456%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/primers/for-use-on~leather-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=primary-cable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32795 pages (at 50 pages/min), scraped 958 items (at 1 items/min) 2025-11-02 09:30:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/setup-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7714N23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227714N23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3822N123&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223822N123%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=primers/color~white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-g-7-phenolic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=primers/for-use-on~leather&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7559498251638&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7559498251638%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mold-release-sprays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robot-wraps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=setup-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/primers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7714N23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227714N23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/primers/for-use-on~leather/ 2025-11-02 09:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shackles/thickness~1-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=molds/material~silica&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:44 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shackles/thickness~1-2-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/printer-cartridges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/printer-carts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9988569181719&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=molds&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9988569181719%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512128%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/privacy-screens/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=primers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7714N23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227714N23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:30:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shackles/thickness~1-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=molybdate-test-kits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shackles/thickness~1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32841 pages (at 46 pages/min), scraped 960 items (at 2 items/min) 2025-11-02 09:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=printer-cartridges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9493236633548&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9493236633548%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001456157%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=printer-carts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=privacy-screens&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:16 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:31:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shackles/thickness~1-2/ 2025-11-02 09:31:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:31:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:31:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~cover/manufacturer~yaskawa-motoman&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/for-use-with~grinders-for-bits-and-burs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-026/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-038/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9985525866263&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=robotics&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9985525866263%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002141177%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002401586%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=monitors/connects-to~hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964947%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=freezer-thermometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mop-handles/performance~nonsparking-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/diameter~3&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=friction-resistant-shoulder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:42 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=robotics/robot-cell-component~cover/manufacturer~yaskawa-motoman&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 09:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9964152645892&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=monitors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9964152645892%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002393955%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4809840438670&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mop%20handles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4809840438670%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002235310%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001538178%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-026inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-038inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-043/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9823T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229823T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:31:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-026inches/ 2025-11-02 09:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-062/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-038inches/ 2025-11-02 09:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=friction-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:07 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:32:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4809840438670&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mop%20handles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4809840438670%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002235310%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32890 pages (at 49 pages/min), scraped 961 items (at 1 items/min) 2025-11-02 09:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=frocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rod-alignment-couplers/thread-size~m20/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/material~rubber-1/material~silicone-rubber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/material~stainless-steel-2/material~17-4-ph-stainless-steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-043inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuel-centrifugal-pumps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-101/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-062inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-043inches/ 2025-11-02 09:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4809840438670&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mop%20handles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4809840438670%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002235310%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-alignment-couplers/thread-size~m20-1-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/material~rubber/material~silicone-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-062inches/ 2025-11-02 09:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=17-4%c2%a0ph-stainless-steel-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3106802242081&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rod%20alignment%20couplers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3106802242081%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333421%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1732707636827&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grommets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1732707636827%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512132%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-101inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/guards/material~steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mountable-cable-ties&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:46 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=17-4%c2%a0ph-stainless-steel-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-101inches/ 2025-11-02 09:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=20-watt-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-ends/shank-thread-size~5-8-11&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=17-4%c2%a0ph-stainless-steel-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=200-watt-light-bulbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 500 Internal Server Error 2025-11-02 09:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8204790604356&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rod%20ends&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8204790604356%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000056749%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:03 [scrapy.spidermiddlewares.httperror] (PID: 148) INFO: Ignoring response <500 https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=17-4%c2%a0ph-stainless-steel-screws&srchtxt=&mixedworld=force&workstationdata=true>: HTTP status code is not handled or not allowed 2025-11-02 09:33:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-ends/shank-thread-size~7-16-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32936 pages (at 46 pages/min), scraped 961 items (at 0 items/min) 2025-11-02 09:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fume-exhausters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rod-ends/shank-thread-size~m4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/guards/material~steel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6071K33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226071K33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mortar/for-use-on~ovens-and-furnaces/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~9-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=funnels/material~porcelain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8204790604356&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rod%20ends&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8204790604356%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013295%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:18 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8204790604356&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rod%20ends&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8204790604356%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000108800%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-122121-helical-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8461832346066&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=funnels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8461832346066%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001512484%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fuse-blocks/fuse-blade-style~maxi/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=59935K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2259935K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=furnace-mortar&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-28125inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1379N41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221379N41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:35 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:35 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/helicoils/military-specification~ms124710/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gum-rubber-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-28125inches/ 2025-11-02 09:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/handles/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuse-blocks/for-fuse-blade-style~maxi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hangers/material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~0-709/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/fuse-blocks/for-fuse-blade-style~maxi/ 2025-11-02 09:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-124710-helical-inserts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-15795-806-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:33:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=handles/material~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuses/current~1000-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hangers/material~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 32980 pages (at 44 pages/min), scraped 964 items (at 3 items/min) 2025-11-02 09:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hastelloy-nickel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~0-709inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/handles/material~brass/ 2025-11-02 09:34:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hats/size~one-size&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~1-375/diameter~1-3-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8931638457760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fuses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8931638457760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001513692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hangers/material~plastic/ 2025-11-02 09:34:17 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~0-709inches/ 2025-11-02 09:34:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hats/size~one-size/ 2025-11-02 09:34:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16996-32-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~1-375inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16997-18-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8931638457760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fuses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8931638457760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001513692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-2656&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-316&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~1-375inches/ 2025-11-02 09:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuses/current~6-1-4-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~2l380/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001557514%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~2-3-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001557303%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8931638457760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fuses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8931638457760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001557538%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fuses/current~801-amp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A128&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A128%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8931638457760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fuses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8931638457760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001513692%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=72035K121&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2272035K121%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuses/current~801-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~12inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3578w-gt-torch-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/drive-size~2-3-8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3009A197&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223009A197%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=2l380-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuses/current~801a&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8931638457760&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=fuses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8931638457760%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001538768%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~12inches/ 2025-11-02 09:35:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/drive-size~2-3-8-1/ 2025-11-02 09:35:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33033 pages (at 53 pages/min), scraped 967 items (at 3 items/min) 2025-11-02 09:35:09 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/military-specification~ms20426ad4-3/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/fuses/current~801a/ 2025-11-02 09:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~20-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rivets/military-specification~ms20426ad6-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~2-45-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4241K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224241K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~2-875/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~4-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-20426-ad4-3-rivets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~0-68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-build-surfaces/backing-type~adhesive&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/drive-size~20-mm-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-20426-ad6-8-rivets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~2-45millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/fuses/manufacturer-equivalent-fuse~gtl/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~2-875inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001550007%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989271180099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20build%20surfaces&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989271180099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539083%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:40 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/drive-size~20-mm-1/ 2025-11-02 09:35:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~2-45millimeters/ 2025-11-02 09:35:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/drive-size~4-1-8-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~2-875inches/ 2025-11-02 09:35:47 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3697N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223697N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7549K88&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227549K88%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/drive-size~4-1-8-1/ 2025-11-02 09:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fuses/manufacturer-equivalent~gtl&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/drive-size~5-mm-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~35-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/drive-size~7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-21043-06-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/fuses/manufacturer-equivalent~gtl/ 2025-11-02 09:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2653472997099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20drivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2653472997099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489912%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~0-028/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33080 pages (at 47 pages/min), scraped 970 items (at 3 items/min) 2025-11-02 09:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~4-469/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~4-1-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2653472997099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20drivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2653472997099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489969%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/locknuts/military-specification~ms21045-7/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~35millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/environment~food-and-beverage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2653472997099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20drivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2653472997099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489912%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~0-028&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~35millimeters/ 2025-11-02 09:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~4-469inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8230254771998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20filaments&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8230254771998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081137%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~4-5inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/material~abs-polycarbonate&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~0-028/ 2025-11-02 09:36:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~4-469inches/ 2025-11-02 09:36:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3483N19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223483N19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~4-5inches/ 2025-11-02 09:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8230254771998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20filaments&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8230254771998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001530644%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/material~aquasys-180&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2653472997099&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hex%20drivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2653472997099%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003489912%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~17-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/material~eco-abs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3665N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223665N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8230254771998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20filaments&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8230254771998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004224832%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~10-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~18-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24671-13-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=fusible-links&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8230254771998&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20filaments&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8230254771998%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001525889%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3664N23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223664N23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001549660%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33127 pages (at 47 pages/min), scraped 972 items (at 2 items/min) 2025-11-02 09:37:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~17-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3653N23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223653N23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~2-5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/3d-printer-filaments/performance~thermally-stable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~18-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~17-32/ 2025-11-02 09:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~12-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A386&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A386%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3664N23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223664N23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~8-9-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~12-2-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~18-mm/ 2025-11-02 09:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/diameter~9-3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001525883%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-filaments/performance-properties~low-thermal-expansion&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~12mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~17-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3664N23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223664N23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~8-9millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/3d-printer-filaments/performance-properties~low-thermal-expansion/ 2025-11-02 09:37:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A415&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A415%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~12mm/ 2025-11-02 09:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/diameter~9-3millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24693-c36-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-24693-c54-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~8-9millimeters/ 2025-11-02 09:37:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:37:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/diameter~9-3millimeters/ 2025-11-02 09:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/material~garolite-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~17mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:02 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A415&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A415%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-nozzles/opening-diameter~1-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33175 pages (at 48 pages/min), scraped 974 items (at 2 items/min) 2025-11-02 09:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~17mm/ 2025-11-02 09:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/material~lead/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=galvanized-steel-vents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9971536404295&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9971536404295%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000165899%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/material~garolite&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-powders/color~white&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~2-5-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~2-4-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A415&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A415%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shafts/material~garolite/ 2025-11-02 09:38:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3695N331&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223695N331%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9999589545328&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20powders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9999589545328%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001536223%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/material~lead&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~9-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548749%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3d-printer-resins/flexibility~rigid&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001551561%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hex-drivers/drive-size~no-10/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7367N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227367N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511250%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5304452195484&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=3d%20printer%20resins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5304452195484%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000096204%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:45 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95090A507&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295090A507%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/thread-size~1-3-8-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2900A243&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222900A243%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screws/military-specification~ms35206-323/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~9-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7482N112&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227482N112%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-drivers/size~no-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~3-25-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:38:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~9-32/ 2025-11-02 09:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hex-drivers/size~no-10/ 2025-11-02 09:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/thread-size~1-3-8-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35333-74-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33226 pages (at 51 pages/min), scraped 978 items (at 4 items/min) 2025-11-02 09:39:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7482N112&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227482N112%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35335-59-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334773%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~3-25mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-35338-135-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shafts/thread-size~m18/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=garden-hose-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:23 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/thread-size~1-3-8-12&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 09:39:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~3-25mm/ 2025-11-02 09:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hex-pipe-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7482N112&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227482N112%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~4-55-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/thread-size~m18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~5-4-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/holders/drill-chuck-mount-type~1-4-hex-shank/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hinge-stops&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51576-12-shoulder-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~5-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334699%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~4-55mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:39:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~5-65-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129226%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~5-4mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=holders/mount-type~quick-change-hex&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lockouts/for-use-with~gas-cylinders/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:02 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~4-55mm/ 2025-11-02 09:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539639%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~6-15-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33272 pages (at 46 pages/min), scraped 978 items (at 0 items/min) 2025-11-02 09:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~5-4mm/ 2025-11-02 09:40:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/holders/mount-type~quick-change-hex/ 2025-11-02 09:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-cylinder-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-19-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~6-3-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129226%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-cylinder-lockouts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=3m-sealants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539639%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~6-15mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-27-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~6-15mm/ 2025-11-02 09:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~6-3mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000129226%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gas-leak-detectors/specifications-met~sae-j2791&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/for-use-on~iron-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539639%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:46 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~6-3mm/ 2025-11-02 09:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/diameter~6-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-36-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6767179521779&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gas%20leak%20detectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6767179521779%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002100815%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3430402569734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3430402569734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929691%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-38-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/machine-screws/military-specification~ms51957-68/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:40:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1130T71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221130T71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51957-45-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/diameter~6mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-saws/diameter~15-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33323 pages (at 51 pages/min), scraped 978 items (at 0 items/min) 2025-11-02 09:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shelf-labels/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shelving/component~shelf-bracket/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/diameter~6mm/ 2025-11-02 09:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-punches/hole-diameter~5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shelving/mounting-location~floor/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~15mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3430402569734&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20punches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3430402569734%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001672908%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51960-63-machine-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shelving/component~shelf-bracket&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shelf-labels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hole-saws/diameter~15mm/ 2025-11-02 09:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51963-22-set-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/material~steel-2/material~w1-tool-steel/tolerance-rating~tight&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8128528462812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8128528462812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002257015%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shelving/mounting-location~floor&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-saws/diameter~50-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951954787837&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rods&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951954787837%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000099520%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510502%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~490j6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1752A71&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221752A71%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8128528462812&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8128528462812%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002107802%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/welding-gas-lenses/tig-industry-number~45v25/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8890K113&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228890K113%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-51968-14-hex-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~50mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/diameter~9-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=4l310-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hole-saws/for-use-on~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rods/thread-size~1-1-8-8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33370 pages (at 47 pages/min), scraped 981 items (at 3 items/min) 2025-11-02 09:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=490j6-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hole-saws/diameter~50mm/ 2025-11-02 09:42:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1173010037349&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hole%20saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1173010037349%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000150026%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=45v25-welding-gas-lenses&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/color-coded-shims/sold-as~assortment/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4298A14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224298A14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hole-saws/for-use-on~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rods/thread-size~1-1-8-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=4l450-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=4l430-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hooks/finish~plastic-coated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hole-saws/for-use-on~stainless-steel/ 2025-11-02 09:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/quick-clamp-hose/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rods/thread-size~1-1-8-8/ 2025-11-02 09:42:36 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gaskets/color~red-brown&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/color-coded-shim-stock-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6392083707491&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hooks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6392083707491%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001552856%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=multiple-voltage-power-supplies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose/fitting-connection~quick-disconnect/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9639739499351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gaskets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9639739499351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002202279%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shims/color-coded-shim-stock-sets/ 2025-11-02 09:42:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shim-stock/sold-as~assortment/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shim-stock/sold-as~assortment/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/connection-style~quick-clamp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=multipurpose-build-your-own-connectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~5l370/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slotted-shims/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/connection-style~quick-clamp/ 2025-11-02 09:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/fitting-connection~quick-disconnect-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33417 pages (at 47 pages/min), scraped 982 items (at 1 items/min) 2025-11-02 09:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/shim-stock-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-094&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose/fitting-connection~quick-disconnect-1/ 2025-11-02 09:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shims/thickness~0-032/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=narrow-mouth-glass-jars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shims/shim-stock-assortments/ 2025-11-02 09:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/slotted-shims-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001505952%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~5v560/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rolled-collars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nas1352c-02-6-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/shims/slotted-shims-7/ 2025-11-02 09:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5s-brooms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5560K656&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225560K656%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/thickness~0-032&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~0-46&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8155835142441&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8155835142441%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001466764%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=5v560-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1352c-02-6-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=natural-fiber-string&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001501081%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=neck-gaiters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:43:53 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roller-chain-attachments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=needle-nose-vise-grips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89895K742&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289895K742%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shipping-drums/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=600-5mm-timing-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33470 pages (at 53 pages/min), scraped 983 items (at 1 items/min) 2025-11-02 09:44:10 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:44:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roller-lever-arms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roller-shafts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=80%252f20-hardware&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shock-cords/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shipping-drums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roller-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-diameter~1-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-diameter~11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-l10-30-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000145040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shock-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a-frame-ladders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000039616%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-diameter~3-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-l20-30-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-l21-20-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1460T14&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221460T14%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000120304%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nema-l21-30-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rollers/roller-material~plastic-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~a49/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~a47/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose/id~9-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6321K38&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226321K38%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33520 pages (at 50 pages/min), scraped 985 items (at 2 items/min) 2025-11-02 09:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a30-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2198398498524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2198398498524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001493279%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-material~plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a49-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-clamps/material~brass-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a47-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~1-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5548K87&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225548K87%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rollers/roller-material~plastic/ 2025-11-02 09:45:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5313593865679&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20clamps&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5313593865679%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000070378%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hose-fittings/fitting-connection~barbed-2/hose-id~3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000125343%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~a86/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~11-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:43 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~12-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000158040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~13-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=network-line-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/connection-style~barbed/for-hose-id~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-fittings/fitting-connection~barbed-2/hose-id~3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000066295%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=a86-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1470T646&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221470T646%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:45:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrasive-blasting-cabinets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000108955%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:01 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hose-fittings/connection-style~barbed/for-hose-id~3-4/ 2025-11-02 09:46:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-nozzles/pipe-size~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1479849910224&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20fittings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1479849910224%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002361852%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001692405%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1470T615&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221470T615%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33572 pages (at 52 pages/min), scraped 988 items (at 3 items/min) 2025-11-02 09:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sight-glass/for-use-with~air/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5364555622820&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5364555622820%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000086699%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-0209/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/hydraulic-reservoirs/for-use-with~hydraulic-fluid/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sight-glass/for-use-with~gasoline/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000108955%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hose-nozzles/pipe-size~3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3404K48&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223404K48%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thin-stock/material~200-201-nickel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5364555622820&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hose%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5364555622820%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009285%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sight-glass/for-use-with~air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-0209&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-reservoirs/oil-outlet-pipe-size~3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydraulic-reservoirs/for-use-with~hydraulic-fluid-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sight-glass/for-use-with~gasoline&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000108955%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6677440772530&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6677440772530%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-0209/ 2025-11-02 09:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=35195K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2235195K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7305746640836&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hydraulic%20reservoirs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7305746640836%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000127781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:43 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/hydraulic-reservoirs/for-use-with~hydraulic-fluid-1/ 2025-11-02 09:46:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nickel-thin-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6677440772530&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6677440772530%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969134%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4791K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224791K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=hydrofluoric-acid-sorbents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5070N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225070N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:46:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indicators/for-use-with~air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=35195K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2235195K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7305746640836&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hydraulic%20reservoirs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7305746640836%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000127781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=indicators/for-use-with~chemicals-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33622 pages (at 50 pages/min), scraped 990 items (at 2 items/min) 2025-11-02 09:47:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979599325048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979599325048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965056%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-037/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4791K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224791K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9979599325048&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=indicators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9979599325048%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179323%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/fitting-connection~claw-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5070N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225070N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/fitting-connection~flanged&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=35195K41&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2235195K41%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7305746640836&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=hydraulic%20reservoirs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7305746640836%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000127781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5070N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225070N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002363819%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002362375%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-0370&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4791K61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224791K61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5577T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225577T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2084T125&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222084T125%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:38 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-0370/ 2025-11-02 09:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5070N15&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225070N15%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=abrasive-stones&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:41 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:47:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/signal-cable/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:50 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:47:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-044/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:56 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5577T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225577T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=signal-cable&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-116/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~0-044/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~ethane-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~lubricant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33672 pages (at 50 pages/min), scraped 992 items (at 2 items/min) 2025-11-02 09:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=absolute-positioning-micrometers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=absorbent-pillows&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-0440&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967533%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5577T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225577T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984502%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-1160&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/for-use-with~oxygen-blend/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:22 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-0440/ 2025-11-02 09:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=79215A667&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2279215A667%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~0-044&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3360N304&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223360N304%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-1160/ 2025-11-02 09:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/silicon-carbide-grinding-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:33 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:48:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~0-044/ 2025-11-02 09:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/for-use-with~oxygen-mixtures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ac-832-2-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silicon-carbide-grinding-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~7-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nipples/for-use-with~oxygen-mixtures/ 2025-11-02 09:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000010293%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rollers/roller-width~9-3-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~npg-20-9&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=accordion-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:56 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000061194%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2278T31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222278T31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/silicone-rubber-cords/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2280523379316&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rollers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2280523379316%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000060332%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002097931%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauge-blocks/gauge-size~25-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7535N128&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227535N128%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33724 pages (at 52 pages/min), scraped 994 items (at 2 items/min) 2025-11-02 09:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=accumulator-chargers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2281T11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222281T11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~p205&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silicone-rubber-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-blocks/size~25-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003916867%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/silver-and-deming-drill-bits/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roof-anchors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~p2bl-115-wf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=roof-drain-grates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauge-blocks/size~25-mm/ 2025-11-02 09:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acetal-homopolymer-round-stock&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/cfm-gauges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002079076%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:41 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauge-ports&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silver-and-deming-drill-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/single-sleeve-compression-tube-branch-tees/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~sc-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:52 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:49:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=no-7-alligator-belt-lacing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002120509%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:49:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/air-meters-and-leak-detectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~sf-31&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=single-sleeve-compression-tube-branch-tees&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/for-use-with~breathable-air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33777 pages (at 53 pages/min), scraped 997 items (at 3 items/min) 2025-11-02 09:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%223395454543301%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acme-cutting-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~sft-22&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonmagnetic-punches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002117317%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sjtw-cords/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964983%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4120K104&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224120K104%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002111686%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7897A57&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227897A57%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~sp-31&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/awls/performance~nonsparking-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002117317%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acme-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sjtw-cords&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002142072%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonsparking-ball-peen-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sledge-hammers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/maximum-pressure~20000-psi-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:45 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonsparking-awls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonsparking-chisels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002117317%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sleeves/environment~food-and-beverage/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=acrylic-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sledge-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/maximum-pressure-psi~20000-psi&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~syh-1-15-16-wf&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33825 pages (at 48 pages/min), scraped 999 items (at 2 items/min) 2025-11-02 09:51:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonsparking-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauges/maximum-pressure-psi~20000-psi/ 2025-11-02 09:51:14 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002120279%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/measures~ac-voltage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=actuators/actuation-type~double-acting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:51:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:51:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sleeves/performance-properties~food-contact&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2773767270124&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gauges&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2773767270124%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002128077%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4982320960919&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=actuators&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4982320960919%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001535849%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sleeves/performance-properties~food-contact/ 2025-11-02 09:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ada-compliant-grab-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-cam-switches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzle-heaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~ucfl204&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:37 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/measures~ac-voltage&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 09:51:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-hammers/manufacturer~bosch&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003916843%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/for-use-with~aluminum-oxide-grit&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insert-bearings/for-mounted-bearing-model-number~ucp211&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slings/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3566500454852&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20hammers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3566500454852%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002270406%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/measurement-unit~beaufort/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/electrical-connection~micro-bnc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tables&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965376%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003916901%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/for-use-on~plastic-1/for-use-on~abs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003916843%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453622%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33878 pages (at 53 pages/min), scraped 999 items (at 0 items/min) 2025-11-02 09:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slings/material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9800267239863&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insulation&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9800267239863%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930351%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929784%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/for-use-with~thick-liquids&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/scale~beaufort&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4441N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224441N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965376%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/slings/material~steel/ 2025-11-02 09:52:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/insulation/for-use-on~plastic-1/for-use-on~polycarbonate/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:23 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969038%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauges/scale~beaufort/ 2025-11-02 09:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3986711649816&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=insert%20bearings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3986711649816%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003916843%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slip-on-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tool-accessories/manufacturer-model-number~3981-01&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/measurement-unit~inches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3282K421&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223282K421%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4441N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224441N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965376%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=insulation/for-use-on~polycarbonate-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1204984202611&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20accessories&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1204984202611%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321491%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tool-chucks/manufacturer~dremel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:44 [py.warnings] (PID: 148) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-02 09:52:45 [scrapy.extensions.feedexport] (PID: 148) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:1 2025-11-02 09:52:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:47 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:52:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:49 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/insulation/for-use-on~polycarbonate-plastic/ 2025-11-02 09:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slip-on-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6504433171172&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20chucks&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6504433171172%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002263710%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/scale~in-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/thermal-dynamics-plasma-cutter-torch-replacement-parts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4441N13&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224441N13%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:52:58 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:53:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tig-torch-nozzles/tig-industry-number~13n09/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tig-torch-nozzles/tig-industry-number~53n59/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:03 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauges/scale~in-/ 2025-11-02 09:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33924 pages (at 46 pages/min), scraped 1000 items (at 1 items/min) 2025-11-02 09:53:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/measurement-unit~kilonewtons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tool-circle-cutters/manufacturer-model-number~4300&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/slot-mills/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-type~acme/material~iron-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/thermal-dynamics-plasma-cutter-torch-replacement-parts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7393463189778&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20circle%20cutters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7393463189778%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002325625%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/tig-torch-nozzles/industry-number~13n09&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tool-cutoff-wheels/for-use-on~stone&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nozzles/tig-torch-nozzles/industry-number~53n59&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9830859981210&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9830859981210%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%229640855677505%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/scale~kn&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:28 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nozzles/tig-torch-nozzles/industry-number~13n09/ 2025-11-02 09:53:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2575572360017&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20cutoff%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2575572360017%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930162%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=slot-mills&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nozzles/tig-torch-nozzles/industry-number~53n59/ 2025-11-02 09:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iron-acme-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=65935A25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2265935A25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:35 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauges/scale~kn/ 2025-11-02 09:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nptf-taps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:46 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotary-tool-cutting-bits/manufacturer-model-number~7800-02&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=iron-check-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/gauges/unit-of-measure~n-cm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/small-parts-grippers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4313338843356&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rotary%20tool%20cutting%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4313338843356%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002321315%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~1-2-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jackets/color~fluorescent-yellow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=jam-nuts/finish~zinc-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334370%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/finish~tin-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gauges/system-of-measurement~cnm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=small-parts-grippers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 33977 pages (at 53 pages/min), scraped 1001 items (at 1 items/min) 2025-11-02 09:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9953543717325&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=jackets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9953543717325%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002196512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=940728682702&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=jam%20nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B940728682702%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/hex-nut-profile~thin-heavy&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/thread-size~7-16-24&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001524896%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/gauges/system-of-measurement~cnm/ 2025-11-02 09:54:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~2-1-4-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000167477%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334370%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451367%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002455177%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:33 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/hex-nut-profile~thin-heavy&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 09:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rotor-magnets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:39 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~2-1-4-8&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 09:54:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/snips/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6432T112&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226432T112%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=keys/material~aluminum-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-bar-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~m15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334370%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453596%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~m20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:54:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/socket-adapter-sets/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=snips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/finishing-plugs/shape~round/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~m48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334646%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=knives/for-use-on~rubber-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34025 pages (at 48 pages/min), scraped 1002 items (at 1 items/min) 2025-11-02 09:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000026064%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002453496%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:15 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~m20&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 09:55:16 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:55:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2846472700070&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=knives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2846472700070%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930459%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-handrail-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=socket-adapter-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-finishing-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:22 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/nut-type~locknut/thread-size~m48&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 09:55:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:29 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/kovar-rounds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9260934905351&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=keys&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9260934905351%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000008512%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/pem-part-number~clss-032-0&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adapters/unit-of-measure~n-m&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002119222%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lags/thread-size~m8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9972745186524&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adapters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9972745186524%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108535%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=latex-rubber-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=round-point-set-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:48 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9974560404121&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lags&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9974560404121%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002333499%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/kovar-rounds/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nuts/thread-size~1-3-4-8-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/socket-weld-tees/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-4-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34075 pages (at 50 pages/min), scraped 1002 items (at 0 items/min) 2025-11-02 09:56:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gib-rails&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rounded-key-assortments&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-back-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-3-4-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002454684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-back-rulers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=socket-weld-tees&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~186-1-0-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:17 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:56:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:56:19 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nuts/thread-size~1-3-4-8/ 2025-11-02 09:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99030A978&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299030A978%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/soft-face-hammers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/mats/finish~adhesive-coated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-767-18&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~1-967-18-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lcn-9140-door-openers%252fclosers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/solder/for-joining~stainless-steel-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lavatory-equipment/manufacturer-model-number~186-1-0-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=router-templates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-removal-grinding-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449088%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002449201%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=soft-face-hammers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-mats&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solder/for-joining~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:03 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 09:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:06 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rtd-thermometers/environment~food-and-beverage&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/solder/for-joining~stainless-steel/ 2025-11-02 09:57:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34124 pages (at 49 pages/min), scraped 1003 items (at 1 items/min) 2025-11-02 09:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesive-spreaders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~3-8-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rtds/environment~wet&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/color~amber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7660621287486&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rtd%20thermometers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7660621287486%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002081137%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rtv-molding-compounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/color~beige&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=LEDs/bulb-shape-number~t6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002450972%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7410228447115&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rtds&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7410228447115%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000018993%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001548139%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3858K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223858K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/color~brown&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002203687%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9948839639468&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=LEDs&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9948839639468%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002068001%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=99030A100&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2299030A100%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6581T51&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226581T51%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001527274%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8607K6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228607K6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/color~dark-gray&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/for-use-on~iron-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~glass-filled-nylon&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rubber/material~rubber-1/material~sbr-recycled-rubber/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539359%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/thread-size~m35&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929566%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/adhesives/for-use-on~titanium-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:57:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/material~rubber-1/material~buna-n-natural-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~iron&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002448469%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glass-mounting-brackets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2364N16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222364N16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~porcelain&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34176 pages (at 52 pages/min), scraped 1007 items (at 4 items/min) 2025-11-02 09:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/soldering-torches/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/material~rubber/material~sbr-recycled-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000164949%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510700%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:11 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/adhesives/for-use-on~iron/ 2025-11-02 09:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929673%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/for-use-on~titanium&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510818%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6684T21&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226684T21%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2364N16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222364N16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/adhesives/for-use-on~titanium/ 2025-11-02 09:58:26 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:58:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=soldering-torches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/formulation~abs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon/material-grade~nylon-6-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/shape~ring&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510818%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092227%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5632811590918&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nylon&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5632811590918%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002327032%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002308142%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2364N16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222364N16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/formulation~butyl-mastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber/shape~straight&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2364N16&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222364N16%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8682K81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228682K81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nylon-finishing-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002093198%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:58:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000124163%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5950211940183&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rubber&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5950211940183%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510818%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:05 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:59:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34229 pages (at 53 pages/min), scraped 1009 items (at 2 items/min) 2025-11-02 09:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9408T146&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229408T146%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adhesives/formulation~pet&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8682K81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228682K81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-coated-rolls&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:18 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/solenoids/shape~wye/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5225163669291&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adhesives&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5225163669291%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002124371%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-compressor-grommets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=locking-swivel-caster&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-conveyor-rollers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7582T25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227582T25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 09:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8682K81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228682K81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solenoids/shape~wye&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-expansion&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-hollow-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-lining&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=168950821422&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B168950821422%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310158%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7582T25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227582T25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~139&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:52 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 09:59:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~143&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-angle-cordless-soldering-irons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2558N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222558N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 09:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/tube-fitting-gaskets/material~rubber-1/environment~sanitary/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000083560%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000116635%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34282 pages (at 53 pages/min), scraped 1011 items (at 2 items/min) 2025-11-02 10:00:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-angle-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7582T25&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227582T25%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K145&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K145%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-angle-vises&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K149&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K149%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=gloves/thickness~0-0065&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-sanitary-tube-fitting-gaskets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2023183652164&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=gloves&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2023183652164%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001466731%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-countersinks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-deadbands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8552T5&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228552T5%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubber-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-handles/thread-size~1-4-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-e-20hp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/for-housing-id~1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-e-05mr&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-es1902&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8767262773158&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=adjustable%20handles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8767262773158%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rubbing-compounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-lb-8023&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/for-housing-id~2-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-housing-id~1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-lb-8504&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/for-housing-id~30-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34334 pages (at 52 pages/min), scraped 1014 items (at 3 items/min) 2025-11-02 10:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-parallels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000155950%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=loctite-silicone-sealants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/for-housing-id~5-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~353&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:19 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-housing-id~30mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~11-1-4-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000061868%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rulers/length~16-ft/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/additive~aluminum-2/lubricant-type~antiseize/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~369&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:33 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/spacers/for-housing-id~30mm/ 2025-11-02 10:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001606931%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K411&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K411%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~374&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=adjustable-size-pin-sockets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000051286%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-housing-id~5-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~19-1-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000104324%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~16-ft-&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/additive~aluminum-2/lubricant-type~antiseize-lubricant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K393&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K393%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009287%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001606931%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001610440%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K398&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K398%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:01:58 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rulers/length~16-ft-/ 2025-11-02 10:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/glue/for-use-on~rubber-2/for-use-on~viton-fluoroelastomer/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001553928%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~2-m&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34385 pages (at 51 pages/min), scraped 1017 items (at 3 items/min) 2025-11-02 10:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009287%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001610440%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001614204%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rulers/overall-length~30-31-64/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/lubricants/lubricant-type~dry-film/additive~graphite-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/for-use-on~viton-fluoroelastomer-rubber&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~412&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001606931%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~24-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:27 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~2-m&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:02:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/glue/for-use-on~viton-fluoroelastomer-rubber/ 2025-11-02 10:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000118417%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001616399%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000009287%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001610440%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~429&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~30-31-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/lubricant-type~dry-film-lubricant/additive~graphite-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:42 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/rulers/length~32-m/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K236&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K236%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000072726%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:47 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rulers/length~30-31-64/ 2025-11-02 10:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003432878%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:55 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:02:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:02:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:02:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:02:56 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~449&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/length~32-m-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=lubricants/trade-name~molykote-g-rapid-plus-paste&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/shaft-diameter~5/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34434 pages (at 49 pages/min), scraped 1018 items (at 1 items/min) 2025-11-02 10:03:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~0151&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/hex-size~9-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000155170%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/rulers/length~32-m-1/ 2025-11-02 10:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=afci%252fgfci-receptacles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=8998767593156&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=lubricants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B8998767593156%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002106005%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=o-rings/dash-number~909&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002104765%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9452K443&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229452K443%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/m8-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/for-shaft-diameter~5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3715417604535&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=o%20rings&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3715417604535%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000080771%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1813A253&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221813A253%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/hex-size~9-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:03:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-actuated-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:32 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/spacers/for-shaft-diameter~5/ 2025-11-02 10:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9751K119&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229751K119%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001694958%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-actuators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/shape~dome/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/magnets/material~stainless-steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-balancers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/m8-connectors/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/thread-size~0-781-32-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~14255&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/thread-size~1-1-4-12/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rulers/overall-length~4-4-m&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/shape~domed&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=magnets/material~stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/thread-size~1-1-8-5-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002093455%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~272&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3043912520240&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=rulers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3043912520240%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004841750%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/spacers/shape~domed/ 2025-11-02 10:04:05 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/magnets/material~stainless-steel/ 2025-11-02 10:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/thread-size~0-781-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34483 pages (at 49 pages/min), scraped 1021 items (at 3 items/min) 2025-11-02 10:04:09 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 10:04:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:09 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-02 10:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002073748%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/thread-size~1-1-4-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1923A22&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221923A22%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=66215A54&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2266215A54%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:16 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/spacers/thread-size~0-781-32/ 2025-11-02 10:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/thread-size~1-1-8-5&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334373%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-compressor-dryers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=maraging-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/spacers/thread-size~1-1-8-5/ 2025-11-02 10:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=office-trash-cans&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=66215A54&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2266215A54%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/thread-size~m200/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=run-capacitors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=runoff-pipe-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rust-inhibiting-lubricants&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:49 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~rw89/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~403&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=66215A54&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2266215A54%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mechanical-springs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-conditioning-hose/for-use-with~r-404a-refrigerant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/thread-size~m200&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spacers/washer-type~spherical/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002080139%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2357252853437&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20conditioning%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2357252853437%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984396%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002451589%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-conditioning-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34534 pages (at 51 pages/min), scraped 1022 items (at 1 items/min) 2025-11-02 10:05:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=rw89-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74765A53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274765A53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-conditioning-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:15 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:05:15 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:05:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1874K11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221874K11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spacers/washer-type~spherical&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-bars&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2632900905686&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2632900905686%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001979382%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74765A53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274765A53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/manufacturer~bostik&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-bins&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sabre-saw-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/manufacturer~tap-magic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saddle-jacks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002245017%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~4799&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:47 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saddle-pipe-couplings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002250048%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=74765A53&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2274765A53%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002138443%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saddle-style-loop-pipe-hangers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7509A816&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227509A816%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/sae-viscosity-grade~50&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/sae-viscosity-grade~5w-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34589 pages (at 55 pages/min), scraped 1023 items (at 1 items/min) 2025-11-02 10:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002250048%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:06:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-cylinders/bore-size~11-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-cutting-band-saw-blades&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003415228%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~break-free-clp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003415241%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spatulas/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4216247920660&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4216247920660%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000094413%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2158K19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222158K19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002088071%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002250048%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~kroil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6861K119&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226861K119%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/signs/message~caution-speed-bump/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1172K85&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221172K85%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-cylinders/linear-air-cylinder-type~round-body&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spatulas&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092622%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4216247920660&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4216247920660%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243608%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sae-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:47 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1463K134&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221463K134%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=speed-bump-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-high-pressure-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-ingots&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:07:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-insulation-jacketing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4216247920660&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4216247920660%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243608%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34641 pages (at 52 pages/min), scraped 1028 items (at 5 items/min) 2025-11-02 10:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-l-channels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~730-uv&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~mobil-dte-extra-heavy&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-lock-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-filter-frames/hvac-filter-trade-size~14-14&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133365%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4216247920660&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20cylinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4216247920660%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002243608%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~mobilcut-100&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002078838%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9074576958623&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20filter%20frames&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9074576958623%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002055875%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7606A62&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227606A62%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~mobilgrease-28&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002136673%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2158K37&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222158K37%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-grips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002124904%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:45 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2507K28&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222507K28%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~mobilux-ep-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-rectangular-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1394K6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221394K6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=6443712110985&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oil&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B6443712110985%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002133577%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil/trade-name~molykote-g-4500-fm-multi-purpose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-flares&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2507K28&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222507K28%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1416K43&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221416K43%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-stairs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34695 pages (at 54 pages/min), scraped 1031 items (at 3 items/min) 2025-11-02 10:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~922&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-filters/hvac-filter-trade-size~12-15&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/oil/trade-name~molykote-g-4500-fm-multi-purpose/ 2025-11-02 10:08:12 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:08:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002085881%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=851279397977&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B851279397977%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002357558%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-glasses/lens-properties~scratch-resistant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2507K28&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222507K28%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-glasses/lens-style~frameless&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7585A44&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227585A44%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/metals/shape~rod-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-filters/hvac-filter-trade-size~15-30&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metal-storage-drums&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=751624917026&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=safety%20glasses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B751624917026%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969238%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=751624917026&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=safety%20glasses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B751624917026%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001973143%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-filters/hvac-filter-trade-size~6-12&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=851279397977&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B851279397977%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002067776%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:43 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/split-dies/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=851279397977&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20filters&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B851279397977%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005043329%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metals/shape~rod-and-disc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oil-separators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=751624917026&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=safety%20glasses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B751624917026%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969238%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/split-ring-tag-fasteners/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=751624917026&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=safety%20glasses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B751624917026%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001973143%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1062337176277&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=metals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1062337176277%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000006299%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-017&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=split-dies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34745 pages (at 50 pages/min), scraped 1033 items (at 2 items/min) 2025-11-02 10:09:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=751624917026&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=safety%20glasses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B751624917026%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001969238%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-strips&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=split-ring-tag-fasteners&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001500391%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=751624917026&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=safety%20glasses&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B751624917026%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001973143%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oilers/for-use-with~lacquer-solvents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=safety-toe-guards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8987K64&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228987K64%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4227901971716&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oilers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4227901971716%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002969031%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:28 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-filtration-systems&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:34 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~0-071&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:36 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oilers/for-use-with~lacquer-solvents&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oilers/for-use-with~slurries/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:40 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:09:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glue/manufacturer-model-number~rtv6708&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oilers/for-use-with~polishing-compounds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sponges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/scales/measurement-unit~kilograms/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/scales/measurement-unit~newtons/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001502606%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spools/material~lead/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092090%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4227901971716&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=oilers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4227901971716%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968111%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5560K76&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225560K76%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:09:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oilers/for-use-with~slurry&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:02 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sponges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scales/scale~kilograms&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scales/scale~newtons&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scissors/for-cutting~rope&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/oilers/for-use-with~slurry/ 2025-11-02 10:10:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34794 pages (at 49 pages/min), scraped 1034 items (at 1 items/min) 2025-11-02 10:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7465T92&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227465T92%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spools/material~lead&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092090%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:12 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/scales/scale~kilograms/ 2025-11-02 10:10:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/scales/scale~newtons/ 2025-11-02 10:10:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5158413542405&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=scissors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5158413542405%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929528%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5560K76&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225560K76%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=45394748479&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B45394748479%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511250%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=metric-t-slotted-framing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/fitting-connection~threaded/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=582295881700&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=glue&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B582295881700%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002092090%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scissors/for-use-on~fabric&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=glycol-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spray-guns/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5560K76&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225560K76%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=millboard-insulation&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5158413542405&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=scissors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5158413542405%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929750%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/connection-style~threaded&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:51 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spray-paint/color~brown/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screw-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:10:55 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/air-hose/connection-style~threaded/ 2025-11-02 10:10:57 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:11:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spray-paint/color~metallic-gold/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34842 pages (at 48 pages/min), scraped 1035 items (at 1 items/min) 2025-11-02 10:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screw-removers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-2-titanium-rods&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-paint/color~brown&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=mobile-device-cases/for-use-with~iphone-13-pro-smartphones&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~1-46&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~1-68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlet-to-raceway-adapters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:21 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5212847902408&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5212847902408%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001527274%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spray-paint/color~metallic-gold&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=moly-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001497876%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9937589136732&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=mobile%20device%20cases&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9937589136732%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004152408%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001494418%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/outlets/current-voltage~2-amp-9v-dc/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7891T3&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227891T3%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5212847902408&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5212847902408%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002192150%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:34 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sandpaper/grit~1800/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6622K19&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226622K19%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7442N31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227442N31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdriver-sets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8989K358&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228989K358%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7596T6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227596T6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/current~2-a-9-v-dc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:52 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outlets/current~300-amp&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/driver-style~l-key&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7442N31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227442N31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:11:59 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/outlets/current~2-a-9-v-dc/ 2025-11-02 10:12:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/air-hose/id~5-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7596T6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227596T6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9978812115307&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=outlets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9978812115307%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001516840%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/ms16624-retaining-rings/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sandpaper/grit~3000&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screwdrivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310948%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34893 pages (at 51 pages/min), scraped 1038 items (at 3 items/min) 2025-11-02 10:12:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprayon-lu-910&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8030K42&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228030K42%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2386623665772&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sandpaper&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2386623665772%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003267605%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:20 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/drive-size~1-5-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/id~5-500mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7596T6&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227596T6%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7442N31&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227442N31%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ms-16624-1150-retaining-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=4952A219&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%224952A219%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screwdrivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310948%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003488415%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/air-hose/id~5-500mm/ 2025-11-02 10:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~186&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~24-1-2-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:41 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:12:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-nuts/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screwdrivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002310948%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001619417%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001594513%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sanitary-clamps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-pullers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose/length~75-ft&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5040K456&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225040K456%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:12:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5040K676&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225040K676%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grade-g-14-garolite-plates&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~1-1-4-7&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:06 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2926966782365&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20hose&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2926966782365%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001624389%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34945 pages (at 52 pages/min), scraped 1041 items (at 3 items/min) 2025-11-02 10:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/spring-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334684%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5138K999&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225138K999%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-pullers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/thread-size~2-56&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=over-the-glasses-safety-goggles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overhead-conveyor-components&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-nozzles/material~316l-stainless-steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=spring-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-nozzles/material~polypropylene&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overhead-door-latches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000031867%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2158951760495&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2158951760495%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510786%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2158951760495&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2158951760495%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001509869%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=graduated-cylinders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nas1352n-4-10-socket-head-cap-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/end-type~side-loop/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=5329K52&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%225329K52%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=granite-stands&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=overlay-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2158951760495&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2158951760495%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510786%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-powered-hammer-retainers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/for-use-on~aluminum-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 34996 pages (at 51 pages/min), scraped 1044 items (at 3 items/min) 2025-11-02 10:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas-1352n-4-10-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/end-type~side-loop&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/finish~zinc-plated/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929899%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-powered-nibblers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2158951760495&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2158951760495%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001510786%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2891885695101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2891885695101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001944261%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/for-use-with~lubricant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=oxygen-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9533K999&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229533K999%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/finish~zinc-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002984502%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas1352n-06-6-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:38 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealants/for-use-with~sodium-chloride&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2891885695101&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2891885695101%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542944%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-slides/bore-size~32-mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nas1352n-6-16-socket-head-cap-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nasm15795/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=681742819075&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=sealants&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B681742819075%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964886%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/washers/specifications-met~nasm15795/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saws/for-use-on~drywall&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7147879609086&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20slides&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7147879609086%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000015066%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-tool-holders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=661583884380&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B661583884380%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930457%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sealers/for-use-on~plastic-1/for-use-on~polystyrene/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6311K481&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226311K481%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nasm-15795-805b-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saws/for-use-on~hastelloy-nickel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35047 pages (at 51 pages/min), scraped 1045 items (at 1 items/min) 2025-11-02 10:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nasm-15795-808b-washers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-wedges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=661583884380&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B661583884380%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002940380%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/saws/for-use-on~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealers/for-use-on~polystyrene-plastic&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sheets/material~natural-gum-foam/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saws/for-use-on~plastic-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aisle-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/sealers/for-use-on~polystyrene-plastic/ 2025-11-02 10:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3054A135&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223054A135%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/alcohol/concentration~91/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sealing-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=661583884380&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=saws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B661583884380%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002930351%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:38 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=saws/for-use-on~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alarm-will-sound-exit-signs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=natural-foam-sheets&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~bonded-polyester-strapping&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/saws/for-use-on~steel/ 2025-11-02 10:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~3-438/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~30/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002965401%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~diesel-fuel-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:03 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=padlock-shrouds&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1958T81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221958T81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35096 pages (at 49 pages/min), scraped 1047 items (at 2 items/min) 2025-11-02 10:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002964782%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=seals/for-use-with~petroleum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~3-438inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pads/material~paper-fiber-buna-n&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~30inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-hose-swivels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3146508506032&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3146508506032%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002967770%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/springs/length~3-438inches/ 2025-11-02 10:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~krytox-gpl-226&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9951811382783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pads&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9951811382783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001539893%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:23 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/springs/length~30inches/ 2025-11-02 10:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~5-75/length~5-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~dri-slide-multi-purpose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grease&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002099901%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~56-8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9556K81&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229556K81%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grease&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002085440%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=10195K151&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2210195K151%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alligator-belt-lacing/lacing-number~20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=scale-ramps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~5-75inches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alligator-belt-lacing/lacing-number~35n&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3797964781943&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=alligator%20belt%20lacing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3797964781943%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000018981%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~56-8millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/springs/length~5-75inches/ 2025-11-02 10:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/length~7-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=security-mirrors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=3797964781943&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=alligator%20belt%20lacing&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B3797964781943%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000141990%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6110K217&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226110K217%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:00 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/springs/length~56-8millimeters/ 2025-11-02 10:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/nipples/material~brass-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pail-lid-closers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~lubriplate-syn-lube-68&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35147 pages (at 51 pages/min), scraped 1050 items (at 3 items/min) 2025-11-02 10:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6118K23&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226118K23%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint/color~dark-green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/length~7millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/springs/material~aluminum-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grease&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002151025%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/material~brass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4999700653188&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=paint&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4999700653188%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002203177%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:24 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/springs/length~7millimeters/ 2025-11-02 10:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1016K91&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221016K91%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~mobil-dte-heavy&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint/color~ford-green&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/nipples/material~brass/ 2025-11-02 10:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminized-gloves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=springs/material~aluminum&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprinkler-valves/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grease&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002080535%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4999700653188&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=paint&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4999700653188%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002199319%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paint/color~hyster-nugget-yellow/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:41 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/springs/material~aluminum/ 2025-11-02 10:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2158K36&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222158K36%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7614T9&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227614T9%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/shaft-diameter~1-2-to-15-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/shaft-diameter~1-2-to-3-4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nipples/pipe-size~6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-nozzles/specifications-met~osha-compliant-29-cfr-1910-95&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-alloy-5005&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprinkler-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:54 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9982017210813&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nipples&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9982017210813%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000047728%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2158951760495&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2158951760495%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108188%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paint/color~hyster-yellow&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/for-shaft-diameter~1-2-to-15-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/for-shaft-diameter~1-2-to-3-4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35199 pages (at 52 pages/min), scraped 1055 items (at 5 items/min) 2025-11-02 10:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/shaft-diameter~1-to-1-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-bar-grating&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:14 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/paint/color~hyster-yellow/ 2025-11-02 10:18:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/for-shaft-diameter~1-2-to-15-16/ 2025-11-02 10:18:15 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/for-shaft-diameter~1-2-to-3-4/ 2025-11-02 10:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2158951760495&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2158951760495%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108188%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=set-screws/thread-size~1-4-20&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/shaft-diameter~13-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/shaft-diameter~16-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nonmetallic-outlet-boxes&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/for-shaft-diameter~1-to-1-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=4940968732154&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=set%20screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B4940968732154%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000013040%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/shaft-diameter~3-8-to-15-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2158951760495&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20nozzles&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2158951760495%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002108188%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:36 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/for-shaft-diameter~1-to-1-7-8/ 2025-11-02 10:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grease/trade-name~synxtreme-hd-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/for-shaft-diameter~13mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-powered-vibrators&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/for-shaft-diameter~16mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/shaft-diameter~7-8-to-1-1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/for-shaft-diameter~3-8-to-15-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7669763092888&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grease&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7669763092888%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002089241%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:50 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/for-shaft-diameter~13mm/ 2025-11-02 10:18:52 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:18:52 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nsf%252fansi-standard-quick-disconnect-tube-fittings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:54 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/for-shaft-diameter~16mm/ 2025-11-02 10:18:54 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/locking-type~lock-washer&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:18:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/for-shaft-diameter~3-8-to-15-16/ 2025-11-02 10:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1821A77&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221821A77%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/for-shaft-diameter~7-8-to-1-1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35249 pages (at 50 pages/min), scraped 1055 items (at 0 items/min) 2025-11-02 10:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5479875482882&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=nuts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5479875482882%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001974552%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-seals/for-use-with~motor-oil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-motors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shaft-seals&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/for-shaft-diameter~7-8-to-1-1-8/ 2025-11-02 10:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-slides/clean-room-standard~fed-std-class-10&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:15 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=nuts/locking-type~lock-washer&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:19:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1157601946191&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shaft%20seals&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1157601946191%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002963074%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pallet-busters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~0-05&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1821A77&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221821A77%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7147879609086&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=air%20slides&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7147879609086%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001485027%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~0-08&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9495772993241&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20plates&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9495772993241%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001465028%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6837N11&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226837N11%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9495772993241&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20plates&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9495772993241%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001469090%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aluminum-plates/thickness~3-000-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=green-light-filters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~13-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89015K128&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289015K128%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1821A77&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221821A77%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=89015K191&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2289015K191%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pallet-top-covers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9495772993241&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20plates&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9495772993241%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001458953%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shafts/tolerance-rating~tight&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/keyway-depth~1-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:19:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-plates/thickness~3mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9008K84&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229008K84%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/keyway-depth~2-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9495772993241&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=aluminum%20plates&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9495772993241%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001467280%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:04 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:20:04 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/keyway-depth~3-8-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=2057766460678&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shafts&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B2057766460678%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000040621%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:06 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/aluminum-plates/thickness~3mm/ 2025-11-02 10:20:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35301 pages (at 52 pages/min), scraped 1058 items (at 3 items/min) 2025-11-02 10:20:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8975K584&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228975K584%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/shears/performance~nonsparking-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/keyway-depth~1-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=air-wheel-casters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/keyway-depth~3-8mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001894984%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/keyway-depth~2mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/oilers/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinders/manufacturer-model-number~763626-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8975K584&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228975K584%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=alcohol-bottles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shears/handle-material~beryllium-copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:34 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/keyway-depth~3-8mm/ 2025-11-02 10:20:37 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/keyway-depth~2mm/ 2025-11-02 10:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sheet-metal-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=7765984901538&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B7765984901538%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002111409%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5847281553902&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=shears&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5847281553902%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002145998%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sheet-stock-oilers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/number-of-teeth~6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=8975K584&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%228975K584%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=panels/material~cellular-glass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/for-use-on~plaster&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-wall-guards&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9962166674953&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=panels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9962166674953%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511173%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/conductivity-sensors/for-testing~oxidation-reduction-potential-orp/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929774%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/number-of-teeth~6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35352 pages (at 51 pages/min), scraped 1059 items (at 1 items/min) 2025-11-02 10:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=amano-time-punch-clocks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ammonia-testers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:10 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:21:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6140K12&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226140K12%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/head-diameter~0-123&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5503183657331&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5503183657331%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001522107%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003710161%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=orp-(oxidation-reduction-potential)-sensors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929774%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/paper/for-use-on~fiberglass-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=outdoor-heaters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/head-diameter~1-1-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchoring-epoxies&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/anchor-type~toggle-eyebolt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001476317%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/sprockets/roller-chain-trade-number~50-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003710161%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929774%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paper/for-use-on~fiberglass&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wide-rim-ring-shims/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=paper-rolls-for-measuring-tool-data-processors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:50 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:21:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9726243375694&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=anchors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9726243375694%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002065849%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:21:57 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/paper/for-use-on~fiberglass/ 2025-11-02 10:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~21-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=91705A352&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2291705A352%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sprockets/roller-chain-trade-size~50-2ss&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003710161%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:06 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:22:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35401 pages (at 49 pages/min), scraped 1060 items (at 1 items/min) 2025-11-02 10:22:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shims/wide-rim-ring-shims-6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/sprockets/roller-chain-trade-size~50-2ss/ 2025-11-02 10:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shipping-trays&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=parking-curbs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:18 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/shims/wide-rim-ring-shims-6/ 2025-11-02 10:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~5-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~21-mm-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/thread-size~8-36&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:24 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/shape-number~w141&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/wire-rope-compression-sleeves/application~not-for-lifting/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/drive-size~21-mm-1/ 2025-11-02 10:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9726243375694&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=anchors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9726243375694%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002334660%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/square-drive-self-drilling-screws/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/shape-number~w146&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000004751771%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~5-32-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9287102609142&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20bits&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9287102609142%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002131601%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=angle-globe-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:45 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/shape-number~w141&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9634K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229634K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anchors/wire-rope-compression-sleeves-not-for-lifting&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:48 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/drive-size~5-32-1/ 2025-11-02 10:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=passivating-fluids&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=angle-grinders/power-source~air&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=square-drive-self-drilling-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:54 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-bits/shape-number~w146&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:22:55 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:22:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9726243375694&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=anchors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9726243375694%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22LEGACYGROUP%22%2C%22values%22%3A%7B%22productSetId%22%3A%222601718594748%22%7D%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shoulder-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9993435663610&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=angle%20grinders&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9993435663610%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005076415%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9634K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229634K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35451 pages (at 50 pages/min), scraped 1061 items (at 1 items/min) 2025-11-02 10:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=3896T61&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%223896T61%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/drive-size~pl2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/squares/shape~foil/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=shutters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=angled-tie-down-rings&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5124285600057&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screwdrivers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5124285600057%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003488387%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pcb-enclosures&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grinding-wheels/abrasive-material~ceramic-alumina-zirconia-alumina-blend-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=9634K32&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%229634K32%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=1533N2&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%221533N2%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:32 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:23:33 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=squares/shape~rod-and-disc&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=squares/shape~foil&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~sr41w/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/abrasive-material~nylon-mesh-aluminum-oxide&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958961006685&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958961006685%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000006299%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958961006685&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958961006685%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313471%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/abrasive-material~ceramic-alumina-zirconia-alumina-blend&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:44 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:23:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silicone-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001515985%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pegboard-hooks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:23:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grinding-wheels/abrasive-material~ceramic-alumina-zirconia-alumina-blend/ 2025-11-02 10:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/batteries/manufacturer-equivalent-number~sr60/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pegboards/material~dry-erase&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958961006685&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958961006685%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313471%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35501 pages (at 50 pages/min), scraped 1063 items (at 2 items/min) 2025-11-02 10:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=506440405265&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pegboards&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B506440405265%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001511766%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=silver-solder&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=anti-siphon-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sr60-batteries&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/for-use-on~cbn-wheels&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=2271A33&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%222271A33%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sink-drain-traps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9958961006685&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9958961006685%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002313471%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=perforated-straps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grinding-wheels/for-use-on~hastelloy-nickel/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002929700%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~15-64-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/aprons/performance~oil-resistant-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=sleeve-retainers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:40 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/for-use-on~hastelloy&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=appliance-timers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grinding-wheels/for-use-with~handheld-disc-sanders&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:51 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grinding-wheels/for-use-on~hastelloy/ 2025-11-02 10:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=perforated-wood&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~15-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aprons/application~oil-resistant&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~ip4/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pestles&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968357%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~m6/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~15-64/ 2025-11-02 10:25:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/aprons/application~oil-resistant/ 2025-11-02 10:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35551 pages (at 50 pages/min), scraped 1064 items (at 1 items/min) 2025-11-02 10:25:08 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-034&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=arc-flash-protection-underwear&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~ip4&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-continuous-hinges/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968357%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001540781%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~m6&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-050&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pfa-plugs&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=aluminum-extrusion-braces&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~ip4/ 2025-11-02 10:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:28 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-034&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:30 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~m6/ 2025-11-02 10:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001516018%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~pls25/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:36 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-continuous-hinges&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9626962926004&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grinding%20wheels&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9626962926004%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002968357%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:39 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screwdrivers/drive-size~t47/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=argon-hose&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:41 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-050&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ph-transmitter&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-087&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-092&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~pls25&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:25:56 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screwdrivers/size~t47&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=phillips-pin-in-head-screwdriver-bits&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001543436%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001537533%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:04 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~pls25/ 2025-11-02 10:26:05 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=snap-acting-switches/csa-file-number~004442-c-000&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:07 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screwdrivers/size~t47/ 2025-11-02 10:26:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35604 pages (at 53 pages/min), scraped 1064 items (at 0 items/min) 2025-11-02 10:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:11 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:11 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-087&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:26:12 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-092&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5714597745783&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=snap%20acting%20switches&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5714597745783%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000005502916%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=as-0420-2-nuts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=photo-bonding-adhesives&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:22 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=7090K215&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%227090K215%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=asme-duct-tape&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:28 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=snap-in-spacers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=snap-locks&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-145&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grippers/material~wood-2&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-149&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001543450%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-hoist-trolleys/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=608445669459&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grippers&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B608445669459%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000128377%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001543110%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/socket-pipe-fitting-plugs/material~steel-2/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-156&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:48 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-145&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:26:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:51 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-149&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:26:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000000156215%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/hole-diameter~1-3-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/finish~nickel-plated&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=astm-f68-copper&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/drive-size~no-14/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-hoist-trolleys&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/hole-diameter~1-7-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=socket-pipe-fitting-plugs/material~steel&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=5875341805879&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=screws&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B5875341805879%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001519486%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=socket-set-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35655 pages (at 51 pages/min), scraped 1065 items (at 1 items/min) 2025-11-02 10:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:10 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/socket-pipe-fitting-plugs/material~steel/ 2025-11-02 10:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/for-hole-diameter~1-3-16&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:17 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/for-hole-diameter~1-7-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:20 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grommets/for-hole-diameter~1-3-16/ 2025-11-02 10:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/drive-size~no-14-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pilot-punches&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/grommets/hole-diameter~19-32/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-pipe-fitting-caps/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:25 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grommets/for-hole-diameter~1-7-8/ 2025-11-02 10:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=soft-tip-set-screws&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:31 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screws/drive-size~no-14-1/ 2025-11-02 10:27:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:27:31 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-2496&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-connectors/number-of-poles~48&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/drive-size~19-mm-1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/screws/drive-size~ip1/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:41 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-365&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001518703%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/for-hole-diameter~19-32&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pin-removal-punches/sold-as~set/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-pipe-fitting-caps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=automatic-oilers&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:48 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=9989162792774&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pin%20connectors&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B9989162792774%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001543842%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001543002%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A791&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A791%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:53 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/grommets/for-hole-diameter~19-32/ 2025-11-02 10:27:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:27:59 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/size~19millimeters&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=screws/size~ip1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=6168T116&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%226168T116%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:02 [scrapy.core.scraper] (PID: 148) ERROR: Spider error processing (referer: https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~0-365&srchtxt=&mixedworld=force&workstationdata=true) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 327, in iter_errback yield next(it) ^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 368, in __next__ return next(self.data) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 379, in return (self._set_referer(r, response) for r in result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 57, in return (r for r in result if self._filter(r, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 54, in return (r for r in result if self._filter(r, response, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 106, in process_sync yield from iterable File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/spiders/mcmaster_carr.py", line 284, in parse_partial_data if table_data.get("partNumber") == sku: ^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get' 2025-11-02 10:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pin-removal-punches/individual-set~set&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35705 pages (at 50 pages/min), scraped 1065 items (at 0 items/min) 2025-11-02 10:28:08 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:08 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screws/size~19millimeters/ 2025-11-02 10:28:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:09 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/screws/size~ip1/ 2025-11-02 10:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A791&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A791%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:13 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pin-removal-punches/individual-set~set/ 2025-11-02 10:28:15 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/diameter~2-5000-mm/diameter~2-500-mm/diameter~2-5-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=grommets/for-use-with~solvents&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:17 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solder/tip-manufacturer-series~pt&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solder/for-station-manufacturer-model-number~wtcps&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:23 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/pins/pem-part-number~tps-187-8/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ax24-v-belts&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1732707636827&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grommets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1732707636827%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179285%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:26 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/solder/tip-manufacturer-series~pt/ 2025-11-02 10:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:27 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/solder/for-station-manufacturer-model-number~wtcps/ 2025-11-02 10:28:28 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/Content/ItmPrsnttnWebPart.aspx?partnbrtxt=95203A791&componentpartnbrtxt=&possiblecompnbrtxt=&attrcompitmids=&attrnm=&attrval=&cntnridtxt=MainContent&proddtllnkclickedInd=true&printprsnttnInd=false&screenDensity=1&cssAlias=undefined&clientNavigationEvents=%5B%7B%22type%22:%22ORDERINGMASTERPARTNUMBER%22,%22selectionType%22:%22SELECTION%22,%22values%22:%7B%22entities%22:%2295203A791%22,%22preselectedComponentPartNumber%22:%22%22%7D%7D%5D&mixedworld=force&workstationdata=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&AttrRelRefinedSort=true&envrmgrcharsetind=3&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solenoid-control-valves&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~2-5mm&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/stainless-steel-square-tubing/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solenoids&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:39 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=solid-rivet-tools&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/captive-pins-5/pem-part-number~tps-187-8&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:42 [scrapy.core.scraper] (PID: 148) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 94, in process_request session_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-02 10:28:42 [mcmaster_carr] (PID: 148) WARNING: no value.entitites: https://www.mcmaster.com/products/pins/diameter~2-5mm/ 2025-11-02 10:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1732707636827&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grommets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1732707636827%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179285%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:46 [scrapy.downloadermiddlewares.retry] (PID: 148) ERROR: Gave up retrying (failed 3 times): 403 response 2025-11-02 10:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/axles/diameter~1-mm/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000002141800%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=pins/diameter~17-64&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:56 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=stainless-steel-square-tubing&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:57 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:28:58 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/v-belts/belt-trade-number~b225/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=315582308815&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=pins&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B315582308815%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000001542583%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/webparts/content/ProdPageWebPart.aspx?cntnridtxt=MainContent&srchidtxt=1732707636827&cntnrwdth=1850&viewporthgt=309&srchrsltdisplovrdind=false&specsrchhexnutsovrdind=false&landingpagesuppressedind=false&srchrslttxt=grommets&fromprodpagewebpart=true&expandedprsnttns=&cssAlias=undefined&clientNavigationEvents=%5B%7B%22values%22%3A%7B%22searchTerms%22%3A%5B1732707636827%5D%7D%2C%22selectionType%22%3A%22SEARCH%22%2C%22type%22%3A%22SEARCHTERM%22%7D%2C%7B%22values%22%3A%7B%22entities%22%3A%5B%2210000003179285%22%5D%7D%2C%22selectionType%22%3A%22SELECTION%22%2C%22type%22%3A%22SPEC%22%7D%5D&ConstructBLPInTileComponents=true&UseNPPImagesHealthCheck=true&UseNPPCopyHealthCheck=true&mixedworld=force&workstationdata=true&ShowPriceBreakSplits=true&usepackquantitypivoting=true&useSpecSearchConversionLogicToggle=true&UseAttrNullsInPubElemSelection=true&CngLineBreaks=true&PDOnlyCNGs=true&UseSellUnitOfMeasureSplits=true&RedesignedRelatedLists=true&EnableItemToProductFamilyProjection=true&ApplyAttrRelRefinements=true&TableRefinementsHorizontalAlignment=true&AttrRelRefinedSort=true&TableRefinementsSpannerAlignmentAndPositioning=true&TableRefinementsTableColumnAlignment=true&LayoutBoxSszs=true&UsePubElementSelectionCache=true&NavigateSearchTermToClickCountsOptimization=true&EnableClientRenderedSpecSearchProxy=true&CheckBLPContentWebPartViewportRemoval=true&useEs6=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=axles/end-type~keyed-1&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/products/axles/diameter~3-16/ already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/tokenauthorization.aspx already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-02 10:29:08 [scrapy.extensions.logstats] (PID: 148) INFO: Crawled 35755 pages (at 50 pages/min), scraped 1066 items (at 1 items/min) 2025-11-02 10:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 148) WARNING: Request https://www.mcmaster.com/mv1761848016/WebParts/RemoteSesnWebPart/SesnAPI.aspx?acttxt=parsehref&href=ground-fault-circuit-protectors&srchtxt=&mixedworld=force&workstationdata=true already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS.